Nov 03 19:36:54.525468 ip-10-0-134-178 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 19:36:54.525482 ip-10-0-134-178 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 19:36:54.525491 ip-10-0-134-178 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 19:36:54.525774 ip-10-0-134-178 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 19:37:04.318650 ip-10-0-134-178 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Nov 03 19:37:04.318665 ip-10-0-134-178 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 237797ecd6e14bfda68abae61361dd56 -- Nov 03 19:38:48.323979 ip-10-0-134-178 systemd[1]: Starting Kubernetes Kubelet... Nov 03 19:38:48.768209 ip-10-0-134-178 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:38:48.768209 ip-10-0-134-178 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 19:38:48.768209 ip-10-0-134-178 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:38:48.768209 ip-10-0-134-178 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:38:48.776140 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776041 2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:38:48.776140 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776134 2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:38:48.776140 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776139 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:38:48.776140 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776143 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:38:48.776140 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776147 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776151 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776157 2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776161 2568 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776164 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776168 2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776172 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776175 2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776179 2568 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776182 2568 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776185 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776189 2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776192 2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776196 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776200 2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776203 2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776207 2568 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776210 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776214 2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776218 2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:38:48.776443 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776221 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776226 2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776230 2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776233 2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776237 2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776241 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776245 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776248 2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776265 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776270 2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776274 2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776277 2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776282 2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776286 2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776292 2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776296 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776301 2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776305 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776309 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:38:48.777267 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776313 2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776317 2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776321 2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776325 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776329 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776334 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776338 2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776343 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776348 2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776352 2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776356 2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776361 2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776365 2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776369 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776373 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776377 2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776382 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776386 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776390 2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:38:48.778068 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776394 2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776398 2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776404 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776408 2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776412 2568 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776416 2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776421 2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776426 2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776432 2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776436 2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776440 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776444 2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776448 2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776452 2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776456 2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776460 2568 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776465 2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776472 2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776480 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776485 2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:38:48.778851 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776490 2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776496 2568 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776500 2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776505 2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776511 2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776517 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776522 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776526 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776530 2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776534 2568 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776538 2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776543 2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776547 2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776551 2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776555 2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776559 2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776564 2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776568 2568 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776573 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776578 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:38:48.779661 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776582 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776587 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776593 2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.776598 2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.777973 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.777983 2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.777988 2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.777992 2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.777997 2568 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778002 2568 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778006 2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778011 2568 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778015 2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778019 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778024 2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778028 2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778033 2568 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778037 2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778041 2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778045 2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:38:48.780319 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778049 2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778053 2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778058 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778062 2568 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778066 2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778070 2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778074 2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778078 2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778082 2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778086 2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778091 2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778095 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778099 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778103 2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778110 2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778115 2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778119 2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778124 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778128 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:38:48.780831 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778132 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778135 2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778139 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778144 2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778149 2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778153 2568 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778157 2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778161 2568 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778166 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778171 2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778175 2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778179 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778183 2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778187 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778192 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778196 2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778200 2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778203 2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778208 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778212 2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:38:48.781568 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778216 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778220 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778224 2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778229 2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778234 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778238 2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778241 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778245 2568 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778250 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778254 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778258 2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778263 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778267 2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778271 2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778275 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778279 2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778283 2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778287 2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778291 2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:38:48.782456 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778295 2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778299 2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778304 2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778309 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778313 2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778317 2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778321 2568 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778325 2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778332 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778336 2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778340 2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778344 2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778348 2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778352 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778356 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778360 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778364 2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778369 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778373 2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:38:48.783258 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778377 2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778381 2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778385 2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778389 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778394 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778398 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778402 2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778406 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778410 2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778414 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778418 2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778422 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.778431 2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779343 2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779354 2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779360 2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779365 2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779369 2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779376 2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:38:48.783752 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779382 2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779387 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779394 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779399 2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779403 2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779407 2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779411 2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779415 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779419 2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779423 2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779428 2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779432 2568 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779437 2568 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779441 2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779445 2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779450 2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779454 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779458 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779463 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779467 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:38:48.784322 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779471 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779475 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779479 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779484 2568 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779488 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779492 2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779496 2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779501 2568 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779505 2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779510 2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779514 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779518 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779522 2568 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779526 2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779532 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779536 2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779540 2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779544 2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779549 2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:38:48.784910 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779553 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779557 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779561 2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779566 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779570 2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779574 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779578 2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779582 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779586 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779590 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779597 2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779603 2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779607 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779612 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779616 2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779620 2568 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779624 2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779628 2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779632 2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:38:48.785379 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779636 2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779640 2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779644 2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779648 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779653 2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779657 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779661 2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779665 2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779670 2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779674 2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779678 2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779682 2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779686 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779690 2568 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779712 2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779717 2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779723 2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779727 2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779731 2568 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779735 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:38:48.785941 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779739 2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779743 2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779747 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779751 2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779755 2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779759 2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779763 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779768 2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779784 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779788 2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779792 2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779796 2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779800 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779804 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779808 2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779812 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779816 2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779821 2568 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779826 2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:38:48.786454 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779830 2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779834 2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.779838 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.780955 2568 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.780973 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.780984 2568 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.780992 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781000 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781005 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781013 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781020 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781026 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781031 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781037 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781042 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781047 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781052 2568 flags.go:64] FLAG: --cgroup-root="" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781057 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781062 2568 flags.go:64] FLAG: --client-ca-file="" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781068 2568 flags.go:64] FLAG: --cloud-provider="external" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781073 2568 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781079 2568 flags.go:64] FLAG: --cluster-domain="" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781084 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 19:38:48.786914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781089 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781094 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781099 2568 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781113 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781118 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781123 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781128 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781133 2568 flags.go:64] FLAG: --contention-profiling="false" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781139 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781144 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781150 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781155 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781162 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781167 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781171 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781176 2568 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781181 2568 flags.go:64] FLAG: --enable-server="true" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781186 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781196 2568 flags.go:64] FLAG: --event-burst="100" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781201 2568 flags.go:64] FLAG: --event-qps="50" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781206 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781211 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781216 2568 flags.go:64] FLAG: --eviction-hard="" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781222 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 19:38:48.787426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781227 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781232 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781237 2568 flags.go:64] FLAG: --eviction-soft="" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781242 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781247 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781252 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781256 2568 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781261 2568 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781266 2568 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781271 2568 flags.go:64] FLAG: --feature-gates="" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781278 2568 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781283 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781288 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781293 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781298 2568 flags.go:64] FLAG: --healthz-port="10248" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781303 2568 flags.go:64] FLAG: --help="false" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781308 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781314 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781319 2568 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781324 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781330 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781335 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781340 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 19:38:48.788022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781344 2568 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781349 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781353 2568 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781360 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781365 2568 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781370 2568 flags.go:64] FLAG: --kube-reserved="" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781375 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781380 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781384 2568 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781389 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781394 2568 flags.go:64] FLAG: --lock-file="" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781398 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781403 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781408 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781417 2568 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781422 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781427 2568 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781431 2568 flags.go:64] FLAG: --logging-format="text" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781436 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781441 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781446 2568 flags.go:64] FLAG: --manifest-url="" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781451 2568 flags.go:64] FLAG: --manifest-url-header="" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781458 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781463 2568 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781469 2568 flags.go:64] FLAG: --max-pods="110" Nov 03 19:38:48.788558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781474 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781479 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781484 2568 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781489 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781494 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781500 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781504 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781515 2568 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781520 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781525 2568 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781531 2568 flags.go:64] FLAG: --pod-cidr="" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781536 2568 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781541 2568 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781545 2568 flags.go:64] FLAG: --pods-per-core="0" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781550 2568 flags.go:64] FLAG: --port="10250" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781554 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781559 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-041e0bb2fea46f955" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781564 2568 flags.go:64] FLAG: --qos-reserved="" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781569 2568 flags.go:64] FLAG: --read-only-port="10255" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781573 2568 flags.go:64] FLAG: --register-node="true" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781578 2568 flags.go:64] FLAG: --register-with-taints="" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781583 2568 flags.go:64] FLAG: --registry-burst="10" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781588 2568 flags.go:64] FLAG: --registry-qps="5" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781592 2568 flags.go:64] FLAG: --reserved-cpus="" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781596 2568 flags.go:64] FLAG: --reserved-memory="" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781603 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 19:38:48.789306 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781608 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781613 2568 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781618 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781623 2568 flags.go:64] FLAG: --runonce="false" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781630 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781634 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781640 2568 flags.go:64] FLAG: --seccomp-default="false" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781644 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781649 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781655 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781661 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781666 2568 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781670 2568 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781675 2568 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781679 2568 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781684 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781691 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781712 2568 flags.go:64] FLAG: --system-cgroups="" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781717 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781726 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781731 2568 flags.go:64] FLAG: --tls-cert-file="" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781735 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781742 2568 flags.go:64] FLAG: --tls-min-version="" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781746 2568 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781751 2568 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 19:38:48.789988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781755 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781760 2568 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781765 2568 flags.go:64] FLAG: --v="2" Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781778 2568 flags.go:64] FLAG: --version="false" Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781785 2568 flags.go:64] FLAG: --vmodule="" Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781791 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.781797 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782044 2568 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782051 2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782056 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782061 2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782067 2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782071 2568 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782076 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782080 2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782085 2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782089 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782094 2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782099 2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782103 2568 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782107 2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:38:48.790558 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782111 2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782115 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782121 2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782125 2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782129 2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782133 2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782138 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782142 2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782146 2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782150 2568 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782154 2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782158 2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782162 2568 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782166 2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782171 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782175 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782179 2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782183 2568 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782188 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782192 2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:38:48.791144 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782196 2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782200 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782205 2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782209 2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782214 2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782218 2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782222 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782226 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782230 2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782235 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782241 2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782247 2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782252 2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782257 2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782263 2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782267 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782272 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782276 2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:38:48.791603 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782280 2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782284 2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782288 2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782292 2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782297 2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782301 2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782308 2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782315 2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782320 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782325 2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782329 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782333 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782338 2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782342 2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782346 2568 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782350 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782356 2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782360 2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782364 2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:38:48.792039 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782368 2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782372 2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782376 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782381 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782385 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782390 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782394 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782398 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782402 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782408 2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782412 2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782416 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782420 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782425 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782429 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782433 2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782437 2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782441 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782445 2568 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:38:48.792466 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782450 2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:38:48.792945 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782453 2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:38:48.792945 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782457 2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:38:48.792945 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782461 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:38:48.792945 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782466 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:38:48.792945 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782470 2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:38:48.792945 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782474 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:38:48.792945 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782478 2568 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:38:48.792945 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782482 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:38:48.792945 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782486 2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:38:48.792945 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782492 2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:38:48.792945 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782496 2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:38:48.792945 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782500 2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:38:48.792945 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782504 2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:38:48.792945 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782508 2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:38:48.792945 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782513 2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:38:48.792945 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.782554 2568 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782621 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782628 2568 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782634 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782638 2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782642 2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782646 2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782650 2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782654 2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782659 2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782663 2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782667 2568 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782672 2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782676 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782680 2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782684 2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782688 2568 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782692 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782713 2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782718 2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:38:48.793335 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782722 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782726 2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782730 2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782734 2568 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782739 2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782743 2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782747 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782752 2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782757 2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782761 2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782765 2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782769 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782773 2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782777 2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782781 2568 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782785 2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782790 2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782795 2568 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782799 2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782803 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782807 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:38:48.793866 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782812 2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782816 2568 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782820 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782824 2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782828 2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782832 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782836 2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782840 2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782845 2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782849 2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782853 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782857 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782860 2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782864 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782871 2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782877 2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782881 2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782886 2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782892 2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:38:48.794363 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782896 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782900 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782904 2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782908 2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782912 2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782917 2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782921 2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782925 2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782929 2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782936 2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782943 2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782949 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782953 2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782957 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782961 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782965 2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782970 2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782974 2568 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782979 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:38:48.794796 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782983 2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782987 2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782991 2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782995 2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.782999 2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783003 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783008 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783012 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783016 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783021 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783025 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783029 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783036 2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783041 2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783045 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783049 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783053 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783057 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783061 2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:38:48.795256 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783066 2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783070 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783074 2568 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783078 2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783083 2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783087 2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783092 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783098 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.783102 2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.790391 2568 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.790405 2568 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.790419 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.790423 2568 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790482 2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790487 2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790490 2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790493 2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790496 2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790499 2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790501 2568 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:38:48.795707 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790504 2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790507 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790509 2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790512 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790514 2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790518 2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790520 2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790523 2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790525 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790528 2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790530 2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790533 2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790536 2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790538 2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790540 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790543 2568 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790548 2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790551 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790554 2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:38:48.796166 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790556 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790559 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790562 2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790565 2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790568 2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790571 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790573 2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790576 2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790579 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790583 2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790587 2568 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790590 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790593 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790597 2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790600 2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790603 2568 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790605 2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790608 2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790611 2568 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:38:48.796592 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790613 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790616 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790618 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790621 2568 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790623 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790626 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790628 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790631 2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790633 2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790636 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790638 2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790641 2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790644 2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790647 2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790649 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790653 2568 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790656 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790659 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790661 2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:38:48.797133 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790664 2568 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790666 2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790669 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790671 2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790674 2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790676 2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790679 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790682 2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790684 2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790687 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790690 2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790692 2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790707 2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790709 2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790712 2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790715 2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790717 2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790720 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790722 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:38:48.797576 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790725 2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790728 2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790732 2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790735 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790737 2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790740 2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790743 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790746 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790748 2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790751 2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790754 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790757 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790759 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790762 2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790765 2568 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790767 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790770 2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790773 2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790775 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790778 2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:38:48.798052 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790780 2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:38:48.798514 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790783 2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:38:48.798514 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790785 2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:38:48.798514 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.790812 2568 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:38:48.798514 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790946 2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:38:48.798514 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790950 2568 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:38:48.798514 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790954 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:38:48.798514 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790956 2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:38:48.798514 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790959 2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:38:48.798514 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790961 2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:38:48.798514 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790964 2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:38:48.798514 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790966 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:38:48.798514 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790969 2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:38:48.798514 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790971 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:38:48.798514 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790974 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:38:48.798514 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790976 2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:38:48.798514 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790979 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790981 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790983 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790987 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790989 2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790992 2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790995 2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.790997 2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791000 2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791002 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791005 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791008 2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791010 2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791013 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791015 2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791018 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791020 2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791022 2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791025 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:38:48.798899 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791027 2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791030 2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791032 2568 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791035 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791037 2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791041 2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791045 2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791047 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791050 2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791053 2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791055 2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791058 2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791060 2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791062 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791064 2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791067 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791069 2568 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791072 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791075 2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:38:48.799328 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791078 2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791081 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791083 2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791085 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791088 2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791090 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791092 2568 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791095 2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791097 2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791100 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791102 2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791104 2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791107 2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791109 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791112 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791114 2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791116 2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791119 2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791121 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:38:48.799766 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791123 2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791126 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791128 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791131 2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791133 2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791136 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791138 2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791141 2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791143 2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791146 2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791148 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791151 2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791154 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791157 2568 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791160 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791162 2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791165 2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791167 2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791169 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791172 2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:38:48.800192 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791175 2568 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:38:48.800652 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791178 2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:38:48.800652 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791182 2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:38:48.800652 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791184 2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:38:48.800652 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791187 2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:38:48.800652 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791189 2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:38:48.800652 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791192 2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:38:48.800652 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791194 2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:38:48.800652 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791197 2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:38:48.800652 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791199 2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:38:48.800652 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791201 2568 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:38:48.800652 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791204 2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:38:48.800652 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791206 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:38:48.800652 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791208 2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:38:48.800652 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791211 2568 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:38:48.800652 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791214 2568 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:38:48.800652 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791216 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.791238 2568 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791274 2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791278 2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791281 2568 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791284 2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791286 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791289 2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791292 2568 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791295 2568 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791297 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791300 2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791302 2568 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791304 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791307 2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791309 2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791311 2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:38:48.801165 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791314 2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791316 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791319 2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791321 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791324 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791326 2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791328 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791331 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791333 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791335 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791338 2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791340 2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791342 2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791345 2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791347 2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791349 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791352 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791354 2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791357 2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:38:48.801868 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791359 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791361 2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791364 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791367 2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791369 2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791372 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791375 2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791377 2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791379 2568 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791382 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791384 2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791388 2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791392 2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791395 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791398 2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791400 2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791406 2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791409 2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791411 2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:38:48.802550 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791413 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791416 2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791418 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791420 2568 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791423 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791426 2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791428 2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791431 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791433 2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791436 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791438 2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791441 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791443 2568 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791445 2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791448 2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791450 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791453 2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791455 2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791457 2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791460 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:38:48.803078 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791463 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791466 2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791468 2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791470 2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791473 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791475 2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791477 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791480 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791482 2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791484 2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791487 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791490 2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791492 2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791494 2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791497 2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791499 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791502 2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791504 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791506 2568 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:38:48.803571 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791509 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791511 2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791514 2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791516 2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791518 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791521 2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791523 2568 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791527 2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791530 2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791536 2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791538 2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791541 2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791545 2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:48.791547 2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.791595 2568 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.793941 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.795075 2568 server.go:1014] "Starting client certificate rotation" Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.795168 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:38:48.804045 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.795205 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:38:48.804457 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.797302 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:38:48.804457 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.800914 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:38:48.812062 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.812044 2568 log.go:25] "Validated CRI v1 runtime API" Nov 03 19:38:48.817396 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.817378 2568 log.go:25] "Validated CRI v1 image API" Nov 03 19:38:48.818222 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.818198 2568 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 19:38:48.820993 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.820972 2568 fs.go:135] Filesystem UUIDs: map[53ee6b7b-c470-46b2-99b5-818e050ccbd0:/dev/nvme0n1p3 78352053-ef6e-48e0-ab23-8945513478cc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 19:38:48.821061 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.820992 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 19:38:48.827394 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.827288 2568 manager.go:217] Machine: {Timestamp:2026-11-03 19:38:48.825938309 +0000 UTC m=+0.376397246 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201378 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f66f32789b18489905980b3f146d2 SystemUUID:ec2f66f3-2789-b184-8990-5980b3f146d2 BootID:237797ec-d6e1-4bfd-a68a-bae61361dd56 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fd:4d:29:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fd:4d:29:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:07:4a:0d:92:d6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 19:38:48.828040 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.828031 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 19:38:48.828153 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.828106 2568 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 19:38:48.829682 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.829665 2568 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 19:38:48.830256 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.830227 2568 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 19:38:48.830388 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.830260 2568 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-178.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 19:38:48.831036 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.831026 2568 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 19:38:48.831101 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.831038 2568 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 19:38:48.831101 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.831064 2568 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:38:48.831101 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.831078 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:38:48.832286 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.832276 2568 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 19:38:48.836556 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.836544 2568 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 19:38:48.836670 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.836662 2568 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 19:38:48.836722 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.836692 2568 kubelet.go:493] "Attempting to sync node with API server" Nov 03 19:38:48.836775 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.836728 2568 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 19:38:48.836775 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.836742 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 19:38:48.836775 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.836771 2568 kubelet.go:405] "Adding apiserver pod source" Nov 03 19:38:48.836854 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.836781 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 19:38:48.837102 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.837090 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:38:48.839791 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.839773 2568 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 19:38:48.847101 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.847084 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:38:48.848030 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.848017 2568 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 19:38:48.848103 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.848046 2568 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 19:38:48.848203 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.848191 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 19:38:48.848256 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.848207 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 19:38:48.848256 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.848213 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 19:38:48.848256 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.848219 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 19:38:48.848256 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.848224 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 19:38:48.848256 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.848230 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 19:38:48.848912 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.848902 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 19:38:48.848912 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.848912 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 19:38:48.848984 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.848920 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 19:38:48.848984 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.848926 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 19:38:48.848984 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.848935 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 19:38:48.848984 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.848944 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 19:38:48.850361 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.850349 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 19:38:48.850398 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.850363 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 19:38:48.854144 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.854131 2568 server.go:1263] "Started kubelet" Nov 03 19:38:48.854267 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.854225 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 19:38:48.854321 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.854258 2568 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 19:38:48.854321 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.854290 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 19:38:48.854804 ip-10-0-134-178 systemd[1]: Started Kubernetes Kubelet. Nov 03 19:38:48.855745 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.855729 2568 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 19:38:48.856924 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.856906 2568 server.go:317] "Adding debug handlers to kubelet server" Nov 03 19:38:48.860556 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:48.860458 2568 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 19:38:48.860940 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.860922 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-178.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 19:38:48.861017 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:48.860993 2568 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-178.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 19:38:48.861070 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:48.861011 2568 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 19:38:48.862088 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:48.860924 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-178.us-east-2.compute.internal.18e4a03b76a1681e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-178.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-178.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:38:48.854104094 +0000 UTC m=+0.404563030,LastTimestamp:2026-11-03 19:38:48.854104094 +0000 UTC m=+0.404563030,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-178.us-east-2.compute.internal,}" Nov 03 19:38:48.863133 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.863118 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 19:38:48.864427 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.864412 2568 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 19:38:48.864598 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:48.864581 2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-178.us-east-2.compute.internal\" not found" Nov 03 19:38:48.865155 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.865138 2568 factory.go:55] Registering systemd factory Nov 03 19:38:48.865155 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.865155 2568 factory.go:223] Registration of the systemd container factory successfully Nov 03 19:38:48.865285 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.865251 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 19:38:48.865285 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.865252 2568 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 19:38:48.865285 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.865269 2568 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 19:38:48.865418 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.865339 2568 factory.go:153] Registering CRI-O factory Nov 03 19:38:48.865418 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.865353 2568 factory.go:223] Registration of the crio container factory successfully Nov 03 19:38:48.865418 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.865388 2568 reconstruct.go:97] "Volume reconstruction finished" Nov 03 19:38:48.865418 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.865395 2568 reconciler.go:29] "Reconciler: start to sync state" Nov 03 19:38:48.865418 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.865399 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 19:38:48.865418 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.865415 2568 factory.go:103] Registering Raw factory Nov 03 19:38:48.865794 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.865427 2568 manager.go:1199] Started watching for new ooms in manager Nov 03 19:38:48.866637 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.866623 2568 manager.go:319] Starting recovery of all containers Nov 03 19:38:48.866773 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:48.866749 2568 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-178.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 19:38:48.874663 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.874638 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6tt8v" Nov 03 19:38:48.879283 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.879265 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6tt8v" Nov 03 19:38:48.880037 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:48.880011 2568 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 19:38:48.880138 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.880097 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 19:38:48.880275 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.880245 2568 manager.go:324] Recovery completed Nov 03 19:38:48.881780 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:48.881750 2568 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 03 19:38:48.884680 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.884663 2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:38:48.886662 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.886647 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:38:48.886771 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.886672 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:38:48.886771 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.886683 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:38:48.887299 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.887284 2568 cpu_manager.go:226] "Starting" policy="none" Nov 03 19:38:48.887299 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.887296 2568 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 19:38:48.887385 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.887313 2568 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 19:38:48.887385 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.887295 2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:38:48.888049 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.888033 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:38:48.888120 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.888064 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:38:48.888120 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:48.888078 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:38:48.965221 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:48.965192 2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-178.us-east-2.compute.internal\" not found" Nov 03 19:38:49.024924 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.024867 2568 policy_none.go:50] "Start" Nov 03 19:38:49.024924 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.024896 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 19:38:49.024924 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.024909 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 19:38:49.026929 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.026918 2568 policy_none.go:44] "Start" Nov 03 19:38:49.055755 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.051293 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 19:38:49.055755 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.051315 2568 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 19:38:49.055755 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.051334 2568 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 19:38:49.055755 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:49.051364 2568 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 19:38:49.055755 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.054918 2568 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:38:49.059928 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.059894 2568 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 19:38:49.062431 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.062419 2568 manager.go:341] "Starting Device Plugin manager" Nov 03 19:38:49.062506 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:49.062462 2568 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 19:38:49.062506 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.062475 2568 server.go:87] "Starting device plugin registration server" Nov 03 19:38:49.062723 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.062710 2568 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 19:38:49.062775 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.062726 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 19:38:49.062843 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.062830 2568 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 19:38:49.062915 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.062905 2568 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 19:38:49.062964 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.062916 2568 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 19:38:49.063430 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:49.063411 2568 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 19:38:49.063525 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:49.063458 2568 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-178.us-east-2.compute.internal\" not found" Nov 03 19:38:49.071658 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:49.071641 2568 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-178.us-east-2.compute.internal\" not found" node="ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.151556 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.151510 2568 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal"] Nov 03 19:38:49.151683 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.151590 2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:38:49.155993 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.155840 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:38:49.156074 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.156008 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:38:49.156074 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.156019 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:38:49.158373 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.158361 2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:38:49.159172 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.159158 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.159224 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.159187 2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:38:49.160271 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.160257 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:38:49.160271 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.160263 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:38:49.160370 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.160286 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:38:49.160370 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.160300 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:38:49.160434 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.160287 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:38:49.160434 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.160429 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:38:49.162426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.162411 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.162495 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.162438 2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:38:49.162774 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.162764 2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:38:49.163308 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.163292 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:38:49.163362 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.163322 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:38:49.163362 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.163336 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:38:49.163472 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.163457 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:38:49.163516 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.163481 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:38:49.163516 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.163495 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:38:49.163516 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.163514 2568 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.167546 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.167529 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b96c57e6fe4ecb21f4910168c569318a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"b96c57e6fe4ecb21f4910168c569318a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.167633 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.167553 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b96c57e6fe4ecb21f4910168c569318a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"b96c57e6fe4ecb21f4910168c569318a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.167633 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.167571 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cf8bc898cbd908afece93ec1ad60d21-config\") pod \"kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"5cf8bc898cbd908afece93ec1ad60d21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.171362 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.171349 2568 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.171409 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:49.171369 2568 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-178.us-east-2.compute.internal\": node \"ip-10-0-134-178.us-east-2.compute.internal\" not found" Nov 03 19:38:49.190265 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:49.190240 2568 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-178.us-east-2.compute.internal\" not found" node="ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.192692 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:49.192676 2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-178.us-east-2.compute.internal\" not found" Nov 03 19:38:49.192786 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:49.192688 2568 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-178.us-east-2.compute.internal\" not found" node="ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.268432 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.268410 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cf8bc898cbd908afece93ec1ad60d21-config\") pod \"kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"5cf8bc898cbd908afece93ec1ad60d21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.268510 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.268435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b96c57e6fe4ecb21f4910168c569318a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"b96c57e6fe4ecb21f4910168c569318a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.268638 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.268450 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b96c57e6fe4ecb21f4910168c569318a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"b96c57e6fe4ecb21f4910168c569318a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.268733 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.268682 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cf8bc898cbd908afece93ec1ad60d21-config\") pod \"kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"5cf8bc898cbd908afece93ec1ad60d21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.268833 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.268813 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b96c57e6fe4ecb21f4910168c569318a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"b96c57e6fe4ecb21f4910168c569318a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.268893 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.268854 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b96c57e6fe4ecb21f4910168c569318a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"b96c57e6fe4ecb21f4910168c569318a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.292851 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:49.292801 2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-178.us-east-2.compute.internal\" not found" Nov 03 19:38:49.393346 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:49.393315 2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-178.us-east-2.compute.internal\" not found" Nov 03 19:38:49.493780 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:49.493753 2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-178.us-east-2.compute.internal\" not found" Nov 03 19:38:49.495918 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.495897 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.498064 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:49.498040 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:49.594830 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:49.594762 2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-178.us-east-2.compute.internal\" not found" Nov 03 19:38:49.695335 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:49.695303 2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-178.us-east-2.compute.internal\" not found" Nov 03 19:38:50.876785 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:50.876761 2568 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:38:50.948095 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:50.948069 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 19:38:50.948490 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:50.948186 2568 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 19:38:50.948490 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:50.948222 2568 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 19:38:50.950219 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:50.950201 2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-178.us-east-2.compute.internal\" not found" Nov 03 19:38:50.981169 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:50.981146 2568 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:38:50.991203 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:50.991185 2568 apiserver.go:52] "Watching apiserver" Nov 03 19:38:51.000661 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.000606 2568 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 19:38:51.000926 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.000907 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-w6vlx","openshift-multus/network-metrics-daemon-z9svh","kube-system/global-pull-secret-syncer-p4t7z","openshift-cluster-node-tuning-operator/tuned-sks78","openshift-image-registry/node-ca-pvs99","openshift-network-operator/iptables-alerter-rhlgs","kube-system/konnectivity-agent-f4x7r","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw","openshift-multus/multus-npg2d","openshift-network-diagnostics/network-check-target-wztqw","openshift-ovn-kubernetes/ovnkube-node-tzzd5","openshift-dns/node-resolver-k666m"] Nov 03 19:38:51.003385 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.003363 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.005344 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.005322 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:38:51.005429 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.005395 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z9svh" podUID="2dcde652-ceea-4a58-9746-eb2087dea702" Nov 03 19:38:51.007416 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.007388 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 19:38:51.007507 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.007459 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rq9xl\"" Nov 03 19:38:51.007507 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.007470 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 19:38:51.007507 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.007481 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:51.007507 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.007489 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 19:38:51.007507 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.007504 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 19:38:51.007717 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.007554 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4t7z" podUID="edae5b46-8158-4dc8-91c9-06ed65126db9" Nov 03 19:38:51.007717 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.007567 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 19:38:51.009560 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.009543 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.011717 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.011701 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:38:51.013925 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.013908 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:38:51.016058 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.016039 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:38:51.017029 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.017014 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 19:38:51.018226 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.018209 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.018448 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.018426 2568 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:51.020431 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.020416 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.022717 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.022700 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:38:51.022788 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.022754 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wztqw" podUID="d85c1bad-d10e-4276-9471-0fbe78bb67b3" Nov 03 19:38:51.024986 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.024969 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.027487 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.027458 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k666m" Nov 03 19:38:51.027722 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.027700 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-cni-multus\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.027828 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.027745 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57r2f\" (UniqueName: \"kubernetes.io/projected/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-kube-api-access-57r2f\") pod \"node-ca-pvs99\" (UID: \"e90fc007-32bc-43c5-99c6-7dbb3f9753d7\") " pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:38:51.027828 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.027778 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysconfig\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.027971 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.027869 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-kubernetes\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.027971 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.027910 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-etc-kubernetes\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.027971 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.027943 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgn5q\" (UniqueName: \"kubernetes.io/projected/44cfa442-7cca-4e81-af9c-789fb885a12c-kube-api-access-dgn5q\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.028123 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.027996 2568 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 19:38:51.028522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.028499 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-sys-fs\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.028907 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.028888 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr8k2\" (UniqueName: \"kubernetes.io/projected/04f022dd-949d-40ab-b00c-902e70ef610e-kube-api-access-gr8k2\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.028985 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.028933 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-multus-certs\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.028985 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.028959 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/edae5b46-8158-4dc8-91c9-06ed65126db9-kubelet-config\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:51.029087 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.028982 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-iptables-alerter-script\") pod \"iptables-alerter-rhlgs\" (UID: \"d9e5c9ec-1940-4cb2-8c55-12da74cfc620\") " pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:38:51.029087 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029006 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44cfa442-7cca-4e81-af9c-789fb885a12c-tmp\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.029087 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029030 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/edae5b46-8158-4dc8-91c9-06ed65126db9-dbus\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:51.029087 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029078 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-system-cni-dir\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.029219 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029118 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.029219 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029149 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-567td\" (UniqueName: \"kubernetes.io/projected/36460aa5-b1f0-4be4-b27b-675c6fecffe1-kube-api-access-567td\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.029219 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029195 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-var-lib-kubelet\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.029313 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029224 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-socket-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.029313 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029272 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-kubelet\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.029399 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029327 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19cae502-56f5-4ec2-9867-205025a68d83-agent-certs\") pod \"konnectivity-agent-f4x7r\" (UID: \"19cae502-56f5-4ec2-9867-205025a68d83\") " pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:38:51.029399 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029366 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.029470 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029402 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-cni-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.029470 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029428 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-hostroot\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.029470 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029450 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-netns\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.029598 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029488 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl29m\" (UniqueName: \"kubernetes.io/projected/398cf284-004e-463a-a74e-0193946df8b7-kube-api-access-zl29m\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.029598 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029508 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysctl-d\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.029598 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029527 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-host\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.029598 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029588 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-cnibin\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.029723 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029622 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19cae502-56f5-4ec2-9867-205025a68d83-konnectivity-ca\") pod \"konnectivity-agent-f4x7r\" (UID: \"19cae502-56f5-4ec2-9867-205025a68d83\") " pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:38:51.029723 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029649 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-socket-dir-parent\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.029723 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029708 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cni-binary-copy\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.029836 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029736 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.029836 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029759 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz9vj\" (UniqueName: \"kubernetes.io/projected/2dcde652-ceea-4a58-9746-eb2087dea702-kube-api-access-jz9vj\") pod \"network-metrics-daemon-z9svh\" (UID: \"2dcde652-ceea-4a58-9746-eb2087dea702\") " pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:38:51.029836 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029798 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg6c5\" (UniqueName: \"kubernetes.io/projected/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-kube-api-access-dg6c5\") pod \"iptables-alerter-rhlgs\" (UID: \"d9e5c9ec-1940-4cb2-8c55-12da74cfc620\") " pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:38:51.029948 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029834 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-systemd\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.029948 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029865 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-cni-bin\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.029948 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029897 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-modprobe-d\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.029948 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029940 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-k8s-cni-cncf-io\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.030090 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029966 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cnibin\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.030090 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.029992 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-registration-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.030090 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030010 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.030090 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030032 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-sys\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.030090 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030060 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-etc-selinux\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.030259 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030093 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-conf-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.030259 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030130 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/398cf284-004e-463a-a74e-0193946df8b7-multus-daemon-config\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.030259 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030165 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-host-slash\") pod \"iptables-alerter-rhlgs\" (UID: \"d9e5c9ec-1940-4cb2-8c55-12da74cfc620\") " pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:38:51.030259 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030190 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-system-cni-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.030259 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030210 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/398cf284-004e-463a-a74e-0193946df8b7-cni-binary-copy\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.030406 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030272 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-host\") pod \"node-ca-pvs99\" (UID: \"e90fc007-32bc-43c5-99c6-7dbb3f9753d7\") " pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:38:51.030406 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030306 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-serviceca\") pod \"node-ca-pvs99\" (UID: \"e90fc007-32bc-43c5-99c6-7dbb3f9753d7\") " pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:38:51.030406 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030330 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-run\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.030406 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030343 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-lib-modules\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.030406 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030382 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs\") pod \"network-metrics-daemon-z9svh\" (UID: \"2dcde652-ceea-4a58-9746-eb2087dea702\") " pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:38:51.030625 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030413 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysctl-conf\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.030625 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030449 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-tuned\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.030625 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030487 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-device-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.030625 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.030525 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:51.033102 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.033088 2568 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:38:51.033149 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.033118 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal"] Nov 03 19:38:51.033149 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.033144 2568 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:38:51.038855 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.038806 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:33:50 +0000 UTC" deadline="2028-05-20 11:31:49.303950264 +0000 UTC" Nov 03 19:38:51.038855 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.038854 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13527h52m58.265099646s" Nov 03 19:38:51.041900 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.041879 2568 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:38:51.042020 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.042005 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal"] Nov 03 19:38:51.048422 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.048402 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-f7kpx" Nov 03 19:38:51.058746 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.058730 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f7kpx" Nov 03 19:38:51.108327 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.108308 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 19:38:51.108405 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.108324 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 19:38:51.108405 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.108322 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 19:38:51.108405 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.108340 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 19:38:51.108405 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.108367 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 19:38:51.108608 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.108418 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7xvgk\"" Nov 03 19:38:51.108608 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.108480 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 19:38:51.108608 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.108520 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 19:38:51.108608 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.108541 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 19:38:51.108901 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.108885 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zxqwg\"" Nov 03 19:38:51.108935 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.108891 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 19:38:51.109158 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109146 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:38:51.109280 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109267 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 19:38:51.109326 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109317 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 19:38:51.109365 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109341 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 19:38:51.109365 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109348 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:38:51.109448 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109433 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 19:38:51.109499 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109459 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 19:38:51.109499 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109493 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 19:38:51.109587 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109498 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 19:38:51.109733 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109714 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 19:38:51.109733 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109729 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 19:38:51.109832 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109734 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 19:38:51.109832 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109716 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zzp2g\"" Nov 03 19:38:51.109924 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109847 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vmwsk\"" Nov 03 19:38:51.109924 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109889 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f2rb8\"" Nov 03 19:38:51.109924 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109894 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xbmvx\"" Nov 03 19:38:51.109924 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.109919 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 19:38:51.110284 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.110270 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rlc2f\"" Nov 03 19:38:51.110325 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.110289 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d4wgq\"" Nov 03 19:38:51.119662 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.119645 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 19:38:51.130850 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.130822 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-host\") pod \"node-ca-pvs99\" (UID: \"e90fc007-32bc-43c5-99c6-7dbb3f9753d7\") " pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:38:51.130951 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.130864 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-serviceca\") pod \"node-ca-pvs99\" (UID: \"e90fc007-32bc-43c5-99c6-7dbb3f9753d7\") " pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:38:51.130951 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.130897 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-run\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.130951 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.130926 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-lib-modules\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.131104 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.130947 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-host\") pod \"node-ca-pvs99\" (UID: \"e90fc007-32bc-43c5-99c6-7dbb3f9753d7\") " pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:38:51.131104 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.130958 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs\") pod \"network-metrics-daemon-z9svh\" (UID: \"2dcde652-ceea-4a58-9746-eb2087dea702\") " pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:38:51.131104 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.130986 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-hosts-file\") pod \"node-resolver-k666m\" (UID: \"29f79e6b-20d5-4ba4-8737-068f0d82b1e6\") " pod="openshift-dns/node-resolver-k666m" Nov 03 19:38:51.131104 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131016 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysctl-conf\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.131104 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131048 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-tuned\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.131104 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131078 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-device-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.131104 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131090 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-run\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.131439 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131114 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:51.131439 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131139 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-cni-multus\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.131439 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131170 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57r2f\" (UniqueName: \"kubernetes.io/projected/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-kube-api-access-57r2f\") pod \"node-ca-pvs99\" (UID: \"e90fc007-32bc-43c5-99c6-7dbb3f9753d7\") " pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:38:51.131439 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131199 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-kubelet\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.131439 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131232 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.131439 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131261 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysconfig\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.131439 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131287 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-kubernetes\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.131439 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131318 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-etc-kubernetes\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.131439 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131348 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-node-log\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.131439 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131378 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgn5q\" (UniqueName: \"kubernetes.io/projected/44cfa442-7cca-4e81-af9c-789fb885a12c-kube-api-access-dgn5q\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.131845 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131444 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-sys-fs\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.131845 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131472 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr8k2\" (UniqueName: \"kubernetes.io/projected/04f022dd-949d-40ab-b00c-902e70ef610e-kube-api-access-gr8k2\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.131845 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131503 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-multus-certs\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.131845 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131553 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/edae5b46-8158-4dc8-91c9-06ed65126db9-kubelet-config\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:51.131845 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131584 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-ovn\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.131845 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131614 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-lib-modules\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.131845 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131642 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-serviceca\") pod \"node-ca-pvs99\" (UID: \"e90fc007-32bc-43c5-99c6-7dbb3f9753d7\") " pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:38:51.131845 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131707 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-ovnkube-script-lib\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.131845 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131739 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-iptables-alerter-script\") pod \"iptables-alerter-rhlgs\" (UID: \"d9e5c9ec-1940-4cb2-8c55-12da74cfc620\") " pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:38:51.131845 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131772 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44cfa442-7cca-4e81-af9c-789fb885a12c-tmp\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.131845 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.131774 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:38:51.131845 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131812 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/edae5b46-8158-4dc8-91c9-06ed65126db9-dbus\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:51.131845 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131841 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-log-socket\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.132175 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131882 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-system-cni-dir\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.132175 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131906 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.132175 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131937 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-567td\" (UniqueName: \"kubernetes.io/projected/36460aa5-b1f0-4be4-b27b-675c6fecffe1-kube-api-access-567td\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.132175 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131971 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-var-lib-kubelet\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.132175 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131995 2568 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 19:38:51.132175 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-socket-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.132175 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132051 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-kubelet\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.132175 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132076 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-ovnkube-config\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.132175 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132125 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19cae502-56f5-4ec2-9867-205025a68d83-agent-certs\") pod \"konnectivity-agent-f4x7r\" (UID: \"19cae502-56f5-4ec2-9867-205025a68d83\") " pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:38:51.132175 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132137 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-device-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.132596 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132200 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.132596 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132307 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.132596 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.132397 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:38:51.132596 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.132445 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret podName:edae5b46-8158-4dc8-91c9-06ed65126db9 nodeName:}" failed. No retries permitted until 2026-11-03 19:38:51.632430556 +0000 UTC m=+2.029054597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret") pod "global-pull-secret-syncer-p4t7z" (UID: "edae5b46-8158-4dc8-91c9-06ed65126db9") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:38:51.132596 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132475 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-cni-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.132596 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132514 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-hostroot\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.132596 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132576 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/754b8c74-0816-45c8-92f6-95725095e262-ovn-node-metrics-cert\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.132857 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132607 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-netns\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.132857 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132623 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl29m\" (UniqueName: \"kubernetes.io/projected/398cf284-004e-463a-a74e-0193946df8b7-kube-api-access-zl29m\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.132857 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132644 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-run-netns\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.132857 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132663 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-var-lib-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.132857 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132708 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-hostroot\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.132857 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132743 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysconfig\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.132857 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132754 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-netns\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.132857 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132802 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-kubernetes\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.133138 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.132872 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-etc-kubernetes\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.133138 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.133035 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysctl-conf\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.133138 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.133082 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs podName:2dcde652-ceea-4a58-9746-eb2087dea702 nodeName:}" failed. No retries permitted until 2026-11-03 19:38:51.633055517 +0000 UTC m=+2.029679544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs") pod "network-metrics-daemon-z9svh" (UID: "2dcde652-ceea-4a58-9746-eb2087dea702") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:38:51.133138 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.133117 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-sys-fs\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.133282 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.133238 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-system-cni-dir\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.133329 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.133287 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-multus-certs\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.133329 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.131491 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-cni-multus\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.133414 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.133329 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/edae5b46-8158-4dc8-91c9-06ed65126db9-kubelet-config\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:51.133459 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.133400 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.133568 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.133491 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-tmp-dir\") pod \"node-resolver-k666m\" (UID: \"29f79e6b-20d5-4ba4-8737-068f0d82b1e6\") " pod="openshift-dns/node-resolver-k666m" Nov 03 19:38:51.133627 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.133584 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysctl-d\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.133674 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.133647 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-host\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.133714 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.133679 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-cnibin\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.133760 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.133712 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.133988 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.133956 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-iptables-alerter-script\") pod \"iptables-alerter-rhlgs\" (UID: \"d9e5c9ec-1940-4cb2-8c55-12da74cfc620\") " pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:38:51.134125 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134035 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19cae502-56f5-4ec2-9867-205025a68d83-konnectivity-ca\") pod \"konnectivity-agent-f4x7r\" (UID: \"19cae502-56f5-4ec2-9867-205025a68d83\") " pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:38:51.134125 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134049 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-kubelet\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.134125 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134087 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-socket-dir-parent\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.134319 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134123 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cni-binary-copy\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.134319 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134173 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.134319 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134239 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz9vj\" (UniqueName: \"kubernetes.io/projected/2dcde652-ceea-4a58-9746-eb2087dea702-kube-api-access-jz9vj\") pod \"network-metrics-daemon-z9svh\" (UID: \"2dcde652-ceea-4a58-9746-eb2087dea702\") " pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:38:51.134319 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134273 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8x2d\" (UniqueName: \"kubernetes.io/projected/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-kube-api-access-f8x2d\") pod \"node-resolver-k666m\" (UID: \"29f79e6b-20d5-4ba4-8737-068f0d82b1e6\") " pod="openshift-dns/node-resolver-k666m" Nov 03 19:38:51.134319 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134306 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg6c5\" (UniqueName: \"kubernetes.io/projected/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-kube-api-access-dg6c5\") pod \"iptables-alerter-rhlgs\" (UID: \"d9e5c9ec-1940-4cb2-8c55-12da74cfc620\") " pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134353 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-systemd\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134383 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-cni-bin\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134437 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-systemd-units\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134494 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-run-ovn-kubernetes\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134524 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-cni-netd\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134613 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-modprobe-d\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134682 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-k8s-cni-cncf-io\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134735 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-socket-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134757 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cnibin\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134815 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-cni-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134838 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19cae502-56f5-4ec2-9867-205025a68d83-konnectivity-ca\") pod \"konnectivity-agent-f4x7r\" (UID: \"19cae502-56f5-4ec2-9867-205025a68d83\") " pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134873 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/edae5b46-8158-4dc8-91c9-06ed65126db9-dbus\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134930 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-socket-dir-parent\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134979 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-systemd\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.134999 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-modprobe-d\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135038 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-cni-bin\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.135194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135061 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-k8s-cni-cncf-io\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135090 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cnibin\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135118 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-env-overrides\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135147 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-cni-bin\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135206 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-registration-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135242 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135271 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-etc-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135290 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cni-binary-copy\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135300 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-sys\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135315 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-registration-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135372 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-etc-selinux\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135445 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-conf-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135474 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-sys\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135489 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/398cf284-004e-463a-a74e-0193946df8b7-multus-daemon-config\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135516 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-slash\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135563 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-systemd\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135594 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7rsm\" (UniqueName: \"kubernetes.io/projected/754b8c74-0816-45c8-92f6-95725095e262-kube-api-access-b7rsm\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.136642 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135622 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k22vq\" (UniqueName: \"kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq\") pod \"network-check-target-wztqw\" (UID: \"d85c1bad-d10e-4276-9471-0fbe78bb67b3\") " pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:38:51.137517 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135767 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.137517 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135948 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.137517 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.136112 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-etc-selinux\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.137517 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.136141 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-conf-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.137517 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.136317 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-var-lib-kubelet\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.137517 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.136613 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-host-slash\") pod \"iptables-alerter-rhlgs\" (UID: \"d9e5c9ec-1940-4cb2-8c55-12da74cfc620\") " pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:38:51.137517 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.136662 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-tuned\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.137517 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.137343 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-host\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.137517 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.137343 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-cnibin\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.137517 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.137385 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysctl-d\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.137517 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.137442 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/398cf284-004e-463a-a74e-0193946df8b7-multus-daemon-config\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.138084 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.137677 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19cae502-56f5-4ec2-9867-205025a68d83-agent-certs\") pod \"konnectivity-agent-f4x7r\" (UID: \"19cae502-56f5-4ec2-9867-205025a68d83\") " pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:38:51.138084 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.135649 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-host-slash\") pod \"iptables-alerter-rhlgs\" (UID: \"d9e5c9ec-1940-4cb2-8c55-12da74cfc620\") " pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:38:51.138084 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.137827 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-system-cni-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.138231 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.138186 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/398cf284-004e-463a-a74e-0193946df8b7-cni-binary-copy\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.138231 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.138207 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-system-cni-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.138468 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.138451 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44cfa442-7cca-4e81-af9c-789fb885a12c-tmp\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.138746 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.138729 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/398cf284-004e-463a-a74e-0193946df8b7-cni-binary-copy\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.141641 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.141617 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgn5q\" (UniqueName: \"kubernetes.io/projected/44cfa442-7cca-4e81-af9c-789fb885a12c-kube-api-access-dgn5q\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.142010 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.141987 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-567td\" (UniqueName: \"kubernetes.io/projected/36460aa5-b1f0-4be4-b27b-675c6fecffe1-kube-api-access-567td\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.142099 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.142060 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57r2f\" (UniqueName: \"kubernetes.io/projected/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-kube-api-access-57r2f\") pod \"node-ca-pvs99\" (UID: \"e90fc007-32bc-43c5-99c6-7dbb3f9753d7\") " pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:38:51.142861 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.142831 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr8k2\" (UniqueName: \"kubernetes.io/projected/04f022dd-949d-40ab-b00c-902e70ef610e-kube-api-access-gr8k2\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.142952 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.142898 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl29m\" (UniqueName: \"kubernetes.io/projected/398cf284-004e-463a-a74e-0193946df8b7-kube-api-access-zl29m\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.143188 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.143170 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz9vj\" (UniqueName: \"kubernetes.io/projected/2dcde652-ceea-4a58-9746-eb2087dea702-kube-api-access-jz9vj\") pod \"network-metrics-daemon-z9svh\" (UID: \"2dcde652-ceea-4a58-9746-eb2087dea702\") " pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:38:51.143341 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.143324 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg6c5\" (UniqueName: \"kubernetes.io/projected/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-kube-api-access-dg6c5\") pod \"iptables-alerter-rhlgs\" (UID: \"d9e5c9ec-1940-4cb2-8c55-12da74cfc620\") " pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:38:51.156194 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.156121 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-npg2d" Nov 03 19:38:51.201242 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.201214 2568 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:38:51.238763 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.238731 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8x2d\" (UniqueName: \"kubernetes.io/projected/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-kube-api-access-f8x2d\") pod \"node-resolver-k666m\" (UID: \"29f79e6b-20d5-4ba4-8737-068f0d82b1e6\") " pod="openshift-dns/node-resolver-k666m" Nov 03 19:38:51.238763 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.238760 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-systemd-units\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.238967 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.238779 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-run-ovn-kubernetes\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.238967 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.238793 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-cni-netd\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.238967 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.238833 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-cni-bin\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.238967 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.238838 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-cni-netd\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.238967 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.238860 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-run-ovn-kubernetes\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.238967 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.238866 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-env-overrides\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.238967 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.238888 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-systemd-units\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.238967 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.238908 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-etc-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.238967 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.238918 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-cni-bin\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.238967 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.238950 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-slash\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.238967 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.238967 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-etc-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.238967 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.238974 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-systemd\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.238996 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7rsm\" (UniqueName: \"kubernetes.io/projected/754b8c74-0816-45c8-92f6-95725095e262-kube-api-access-b7rsm\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239019 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22vq\" (UniqueName: \"kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq\") pod \"network-check-target-wztqw\" (UID: \"d85c1bad-d10e-4276-9471-0fbe78bb67b3\") " pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239021 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-slash\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239028 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-systemd\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239068 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-hosts-file\") pod \"node-resolver-k666m\" (UID: \"29f79e6b-20d5-4ba4-8737-068f0d82b1e6\") " pod="openshift-dns/node-resolver-k666m" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239106 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-kubelet\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239127 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239152 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-node-log\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239160 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-hosts-file\") pod \"node-resolver-k666m\" (UID: \"29f79e6b-20d5-4ba4-8737-068f0d82b1e6\") " pod="openshift-dns/node-resolver-k666m" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239206 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-kubelet\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239239 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-ovn\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239262 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-node-log\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239269 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-ovnkube-script-lib\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239208 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239272 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-ovn\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239299 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-log-socket\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239376 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-log-socket\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.239522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239401 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-ovnkube-config\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.240216 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239424 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/754b8c74-0816-45c8-92f6-95725095e262-ovn-node-metrics-cert\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.240216 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239465 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-run-netns\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.240216 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239507 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-run-netns\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.240216 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-var-lib-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.240216 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239569 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-tmp-dir\") pod \"node-resolver-k666m\" (UID: \"29f79e6b-20d5-4ba4-8737-068f0d82b1e6\") " pod="openshift-dns/node-resolver-k666m" Nov 03 19:38:51.240216 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239587 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.240216 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239637 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-var-lib-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.240216 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239648 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.240216 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.239991 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-env-overrides\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.240216 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.240016 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-tmp-dir\") pod \"node-resolver-k666m\" (UID: \"29f79e6b-20d5-4ba4-8737-068f0d82b1e6\") " pod="openshift-dns/node-resolver-k666m" Nov 03 19:38:51.240216 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.240191 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-ovnkube-config\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.240556 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.240168 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-ovnkube-script-lib\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.241559 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.241526 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/754b8c74-0816-45c8-92f6-95725095e262-ovn-node-metrics-cert\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.245216 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.245194 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:38:51.245216 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.245213 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:38:51.245216 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.245223 2568 projected.go:196] Error preparing data for projected volume kube-api-access-k22vq for pod openshift-network-diagnostics/network-check-target-wztqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:38:51.245448 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.245281 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq podName:d85c1bad-d10e-4276-9471-0fbe78bb67b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:38:51.745264575 +0000 UTC m=+2.141888616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k22vq" (UniqueName: "kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq") pod "network-check-target-wztqw" (UID: "d85c1bad-d10e-4276-9471-0fbe78bb67b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:38:51.247109 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.247089 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7rsm\" (UniqueName: \"kubernetes.io/projected/754b8c74-0816-45c8-92f6-95725095e262-kube-api-access-b7rsm\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.247248 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.247226 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8x2d\" (UniqueName: \"kubernetes.io/projected/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-kube-api-access-f8x2d\") pod \"node-resolver-k666m\" (UID: \"29f79e6b-20d5-4ba4-8737-068f0d82b1e6\") " pod="openshift-dns/node-resolver-k666m" Nov 03 19:38:51.314630 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:51.314574 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf8bc898cbd908afece93ec1ad60d21.slice/crio-e3f85f4357a1080a52b18f3e46597fe4fc99fbfc0b8ef0c36127d7c2456a032e WatchSource:0}: Error finding container e3f85f4357a1080a52b18f3e46597fe4fc99fbfc0b8ef0c36127d7c2456a032e: Status 404 returned error can't find the container with id e3f85f4357a1080a52b18f3e46597fe4fc99fbfc0b8ef0c36127d7c2456a032e Nov 03 19:38:51.315123 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:51.315103 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96c57e6fe4ecb21f4910168c569318a.slice/crio-929b901e4ac2e85a87003ac7eab25b79fce54e164c68cd1037616184ac92e757 WatchSource:0}: Error finding container 929b901e4ac2e85a87003ac7eab25b79fce54e164c68cd1037616184ac92e757: Status 404 returned error can't find the container with id 929b901e4ac2e85a87003ac7eab25b79fce54e164c68cd1037616184ac92e757 Nov 03 19:38:51.319487 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.319470 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:38:51.345865 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.345848 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:38:51.349001 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.348986 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:38:51.355358 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:51.355337 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36460aa5_b1f0_4be4_b27b_675c6fecffe1.slice/crio-98c7739fe14600d71e46476a02d21abd6835ff981f203b953d3ed45564bcc735 WatchSource:0}: Error finding container 98c7739fe14600d71e46476a02d21abd6835ff981f203b953d3ed45564bcc735: Status 404 returned error can't find the container with id 98c7739fe14600d71e46476a02d21abd6835ff981f203b953d3ed45564bcc735 Nov 03 19:38:51.357130 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:51.357107 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44cfa442_7cca_4e81_af9c_789fb885a12c.slice/crio-fe236dc97a3c739a1154407f4b75f3c6c9c563dd2c702cd7c807a2fd08da9923 WatchSource:0}: Error finding container fe236dc97a3c739a1154407f4b75f3c6c9c563dd2c702cd7c807a2fd08da9923: Status 404 returned error can't find the container with id fe236dc97a3c739a1154407f4b75f3c6c9c563dd2c702cd7c807a2fd08da9923 Nov 03 19:38:51.368093 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.368078 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:38:51.376409 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:51.376388 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90fc007_32bc_43c5_99c6_7dbb3f9753d7.slice/crio-d098744600485db73bc999ffce2fe5f74e8d7db95a1fca5ea601428d7183754f WatchSource:0}: Error finding container d098744600485db73bc999ffce2fe5f74e8d7db95a1fca5ea601428d7183754f: Status 404 returned error can't find the container with id d098744600485db73bc999ffce2fe5f74e8d7db95a1fca5ea601428d7183754f Nov 03 19:38:51.385926 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.385908 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:38:51.393644 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:51.393621 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e5c9ec_1940_4cb2_8c55_12da74cfc620.slice/crio-e9294007165b9536e5b6c299bde0b1c55811e564c02ba55ffba4f109e7466882 WatchSource:0}: Error finding container e9294007165b9536e5b6c299bde0b1c55811e564c02ba55ffba4f109e7466882: Status 404 returned error can't find the container with id e9294007165b9536e5b6c299bde0b1c55811e564c02ba55ffba4f109e7466882 Nov 03 19:38:51.405840 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.405824 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:38:51.412971 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:51.412947 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19cae502_56f5_4ec2_9867_205025a68d83.slice/crio-46c0a52de1d4ee71e4df2bf1446e536a5c180b6ef54aa5fd211de35d28bc33bb WatchSource:0}: Error finding container 46c0a52de1d4ee71e4df2bf1446e536a5c180b6ef54aa5fd211de35d28bc33bb: Status 404 returned error can't find the container with id 46c0a52de1d4ee71e4df2bf1446e536a5c180b6ef54aa5fd211de35d28bc33bb Nov 03 19:38:51.437777 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.437756 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:38:51.446917 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:51.446895 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f022dd_949d_40ab_b00c_902e70ef610e.slice/crio-0bfcbddc6f3264cec0e1bda6f9f7f985c0c4f55ae5c1f60aac594851ca3192da WatchSource:0}: Error finding container 0bfcbddc6f3264cec0e1bda6f9f7f985c0c4f55ae5c1f60aac594851ca3192da: Status 404 returned error can't find the container with id 0bfcbddc6f3264cec0e1bda6f9f7f985c0c4f55ae5c1f60aac594851ca3192da Nov 03 19:38:51.481207 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.481184 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k666m" Nov 03 19:38:51.487946 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.487929 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:38:51.489917 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:51.489895 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f79e6b_20d5_4ba4_8737_068f0d82b1e6.slice/crio-5bedb955f7c772917b34a2abd401baf9ac3d269821138b52552337fbb48b3ec6 WatchSource:0}: Error finding container 5bedb955f7c772917b34a2abd401baf9ac3d269821138b52552337fbb48b3ec6: Status 404 returned error can't find the container with id 5bedb955f7c772917b34a2abd401baf9ac3d269821138b52552337fbb48b3ec6 Nov 03 19:38:51.496502 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:51.496481 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754b8c74_0816_45c8_92f6_95725095e262.slice/crio-b8259f0b1faea7f133349d60ba4dbdc55499f06e5c55e05afbf25da5003d6b1d WatchSource:0}: Error finding container b8259f0b1faea7f133349d60ba4dbdc55499f06e5c55e05afbf25da5003d6b1d: Status 404 returned error can't find the container with id b8259f0b1faea7f133349d60ba4dbdc55499f06e5c55e05afbf25da5003d6b1d Nov 03 19:38:51.571401 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:38:51.571374 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398cf284_004e_463a_a74e_0193946df8b7.slice/crio-961dfe6bc26775e1420659881bf84559180ac73200913856e80d0d3cf3468246 WatchSource:0}: Error finding container 961dfe6bc26775e1420659881bf84559180ac73200913856e80d0d3cf3468246: Status 404 returned error can't find the container with id 961dfe6bc26775e1420659881bf84559180ac73200913856e80d0d3cf3468246 Nov 03 19:38:51.643128 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.643099 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs\") pod \"network-metrics-daemon-z9svh\" (UID: \"2dcde652-ceea-4a58-9746-eb2087dea702\") " pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:38:51.643314 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.643134 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:51.643314 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.643252 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:38:51.643314 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.643311 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs podName:2dcde652-ceea-4a58-9746-eb2087dea702 nodeName:}" failed. No retries permitted until 2026-11-03 19:38:52.643298095 +0000 UTC m=+3.039922137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs") pod "network-metrics-daemon-z9svh" (UID: "2dcde652-ceea-4a58-9746-eb2087dea702") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:38:51.643433 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.643252 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:38:51.643433 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.643385 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret podName:edae5b46-8158-4dc8-91c9-06ed65126db9 nodeName:}" failed. No retries permitted until 2026-11-03 19:38:52.643372826 +0000 UTC m=+3.039996868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret") pod "global-pull-secret-syncer-p4t7z" (UID: "edae5b46-8158-4dc8-91c9-06ed65126db9") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:38:51.844058 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:51.843994 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22vq\" (UniqueName: \"kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq\") pod \"network-check-target-wztqw\" (UID: \"d85c1bad-d10e-4276-9471-0fbe78bb67b3\") " pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:38:51.844210 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.844126 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:38:51.844210 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.844142 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:38:51.844210 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.844151 2568 projected.go:196] Error preparing data for projected volume kube-api-access-k22vq for pod openshift-network-diagnostics/network-check-target-wztqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:38:51.844210 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:51.844199 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq podName:d85c1bad-d10e-4276-9471-0fbe78bb67b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:38:52.844186964 +0000 UTC m=+3.240810992 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k22vq" (UniqueName: "kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq") pod "network-check-target-wztqw" (UID: "d85c1bad-d10e-4276-9471-0fbe78bb67b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:38:52.062634 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.062489 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:33:51 +0000 UTC" deadline="2028-05-28 10:22:01.669474791 +0000 UTC" Nov 03 19:38:52.062634 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.062563 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13718h43m9.606916961s" Nov 03 19:38:52.208430 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.207933 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:38:52.208430 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:52.208058 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wztqw" podUID="d85c1bad-d10e-4276-9471-0fbe78bb67b3" Nov 03 19:38:52.216931 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.216846 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k666m" event={"ID":"29f79e6b-20d5-4ba4-8737-068f0d82b1e6","Type":"ContainerStarted","Data":"5bedb955f7c772917b34a2abd401baf9ac3d269821138b52552337fbb48b3ec6"} Nov 03 19:38:52.231240 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.231213 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" event={"ID":"04f022dd-949d-40ab-b00c-902e70ef610e","Type":"ContainerStarted","Data":"0bfcbddc6f3264cec0e1bda6f9f7f985c0c4f55ae5c1f60aac594851ca3192da"} Nov 03 19:38:52.238987 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.238939 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rhlgs" event={"ID":"d9e5c9ec-1940-4cb2-8c55-12da74cfc620","Type":"ContainerStarted","Data":"e9294007165b9536e5b6c299bde0b1c55811e564c02ba55ffba4f109e7466882"} Nov 03 19:38:52.249114 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.249087 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pvs99" event={"ID":"e90fc007-32bc-43c5-99c6-7dbb3f9753d7","Type":"ContainerStarted","Data":"d098744600485db73bc999ffce2fe5f74e8d7db95a1fca5ea601428d7183754f"} Nov 03 19:38:52.263693 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.263665 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6vlx" event={"ID":"36460aa5-b1f0-4be4-b27b-675c6fecffe1","Type":"ContainerStarted","Data":"98c7739fe14600d71e46476a02d21abd6835ff981f203b953d3ed45564bcc735"} Nov 03 19:38:52.269979 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.269949 2568 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:38:52.277281 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.277200 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" event={"ID":"5cf8bc898cbd908afece93ec1ad60d21","Type":"ContainerStarted","Data":"e3f85f4357a1080a52b18f3e46597fe4fc99fbfc0b8ef0c36127d7c2456a032e"} Nov 03 19:38:52.295524 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.295453 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-npg2d" event={"ID":"398cf284-004e-463a-a74e-0193946df8b7","Type":"ContainerStarted","Data":"961dfe6bc26775e1420659881bf84559180ac73200913856e80d0d3cf3468246"} Nov 03 19:38:52.305952 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.305917 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" event={"ID":"754b8c74-0816-45c8-92f6-95725095e262","Type":"ContainerStarted","Data":"b8259f0b1faea7f133349d60ba4dbdc55499f06e5c55e05afbf25da5003d6b1d"} Nov 03 19:38:52.315757 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.315729 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f4x7r" event={"ID":"19cae502-56f5-4ec2-9867-205025a68d83","Type":"ContainerStarted","Data":"46c0a52de1d4ee71e4df2bf1446e536a5c180b6ef54aa5fd211de35d28bc33bb"} Nov 03 19:38:52.318146 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.318109 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sks78" event={"ID":"44cfa442-7cca-4e81-af9c-789fb885a12c","Type":"ContainerStarted","Data":"fe236dc97a3c739a1154407f4b75f3c6c9c563dd2c702cd7c807a2fd08da9923"} Nov 03 19:38:52.327522 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.327497 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" event={"ID":"b96c57e6fe4ecb21f4910168c569318a","Type":"ContainerStarted","Data":"929b901e4ac2e85a87003ac7eab25b79fce54e164c68cd1037616184ac92e757"} Nov 03 19:38:52.651695 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.651619 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs\") pod \"network-metrics-daemon-z9svh\" (UID: \"2dcde652-ceea-4a58-9746-eb2087dea702\") " pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:38:52.651695 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.651669 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:52.651916 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:52.651846 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:38:52.651916 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:52.651904 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret podName:edae5b46-8158-4dc8-91c9-06ed65126db9 nodeName:}" failed. No retries permitted until 2026-11-03 19:38:54.651886197 +0000 UTC m=+5.048510228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret") pod "global-pull-secret-syncer-p4t7z" (UID: "edae5b46-8158-4dc8-91c9-06ed65126db9") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:38:52.652329 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:52.652312 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:38:52.652472 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:52.652403 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs podName:2dcde652-ceea-4a58-9746-eb2087dea702 nodeName:}" failed. No retries permitted until 2026-11-03 19:38:54.652391348 +0000 UTC m=+5.049015376 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs") pod "network-metrics-daemon-z9svh" (UID: "2dcde652-ceea-4a58-9746-eb2087dea702") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:38:52.853743 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:52.853709 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22vq\" (UniqueName: \"kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq\") pod \"network-check-target-wztqw\" (UID: \"d85c1bad-d10e-4276-9471-0fbe78bb67b3\") " pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:38:52.853905 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:52.853883 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:38:52.853905 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:52.853900 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:38:52.854003 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:52.853912 2568 projected.go:196] Error preparing data for projected volume kube-api-access-k22vq for pod openshift-network-diagnostics/network-check-target-wztqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:38:52.854003 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:52.853966 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq podName:d85c1bad-d10e-4276-9471-0fbe78bb67b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:38:54.85394822 +0000 UTC m=+5.250572261 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k22vq" (UniqueName: "kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq") pod "network-check-target-wztqw" (UID: "d85c1bad-d10e-4276-9471-0fbe78bb67b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:38:53.063724 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:53.063610 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:33:51 +0000 UTC" deadline="2028-08-01 15:19:37.274081194 +0000 UTC" Nov 03 19:38:53.063724 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:53.063672 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15283h40m44.210413119s" Nov 03 19:38:53.205692 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:53.205660 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:53.205868 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:53.205788 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4t7z" podUID="edae5b46-8158-4dc8-91c9-06ed65126db9" Nov 03 19:38:53.206187 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:53.206167 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:38:53.206290 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:53.206270 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z9svh" podUID="2dcde652-ceea-4a58-9746-eb2087dea702" Nov 03 19:38:54.037675 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:54.037641 2568 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:38:54.206223 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:54.206191 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:38:54.206807 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:54.206322 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wztqw" podUID="d85c1bad-d10e-4276-9471-0fbe78bb67b3" Nov 03 19:38:54.671980 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:54.671233 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs\") pod \"network-metrics-daemon-z9svh\" (UID: \"2dcde652-ceea-4a58-9746-eb2087dea702\") " pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:38:54.671980 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:54.671281 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:54.671980 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:54.671421 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:38:54.671980 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:54.671477 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret podName:edae5b46-8158-4dc8-91c9-06ed65126db9 nodeName:}" failed. No retries permitted until 2026-11-03 19:38:58.671459266 +0000 UTC m=+9.068083312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret") pod "global-pull-secret-syncer-p4t7z" (UID: "edae5b46-8158-4dc8-91c9-06ed65126db9") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:38:54.671980 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:54.671864 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:38:54.671980 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:54.671906 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs podName:2dcde652-ceea-4a58-9746-eb2087dea702 nodeName:}" failed. No retries permitted until 2026-11-03 19:38:58.671893484 +0000 UTC m=+9.068517519 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs") pod "network-metrics-daemon-z9svh" (UID: "2dcde652-ceea-4a58-9746-eb2087dea702") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:38:54.876148 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:54.876081 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22vq\" (UniqueName: \"kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq\") pod \"network-check-target-wztqw\" (UID: \"d85c1bad-d10e-4276-9471-0fbe78bb67b3\") " pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:38:54.876860 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:54.876361 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:38:54.876860 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:54.876385 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:38:54.876860 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:54.876398 2568 projected.go:196] Error preparing data for projected volume kube-api-access-k22vq for pod openshift-network-diagnostics/network-check-target-wztqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:38:54.876860 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:54.876460 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq podName:d85c1bad-d10e-4276-9471-0fbe78bb67b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:38:58.876441192 +0000 UTC m=+9.273065241 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k22vq" (UniqueName: "kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq") pod "network-check-target-wztqw" (UID: "d85c1bad-d10e-4276-9471-0fbe78bb67b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:38:55.210784 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:55.210747 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:55.211256 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:55.210906 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4t7z" podUID="edae5b46-8158-4dc8-91c9-06ed65126db9" Nov 03 19:38:55.211425 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:55.211345 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:38:55.211515 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:55.211459 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z9svh" podUID="2dcde652-ceea-4a58-9746-eb2087dea702" Nov 03 19:38:55.358867 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:55.358835 2568 generic.go:356] "Generic (PLEG): container finished" podID="36460aa5-b1f0-4be4-b27b-675c6fecffe1" containerID="76daf4917c1d96eb4e4198fbe17526941397a4cf2938c711b49f7224257d93fc" exitCode=0 Nov 03 19:38:55.359000 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:55.358944 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6vlx" event={"ID":"36460aa5-b1f0-4be4-b27b-675c6fecffe1","Type":"ContainerDied","Data":"76daf4917c1d96eb4e4198fbe17526941397a4cf2938c711b49f7224257d93fc"} Nov 03 19:38:56.206910 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:56.206371 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:38:56.206910 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:56.206487 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wztqw" podUID="d85c1bad-d10e-4276-9471-0fbe78bb67b3" Nov 03 19:38:56.370025 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:56.369968 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f4x7r" event={"ID":"19cae502-56f5-4ec2-9867-205025a68d83","Type":"ContainerStarted","Data":"31878613cf4448965cca8c7c7b2ce8688ab10ea0ad7cd5f0ee5ac2fde668cd87"} Nov 03 19:38:56.377898 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:56.377868 2568 generic.go:356] "Generic (PLEG): container finished" podID="b96c57e6fe4ecb21f4910168c569318a" containerID="a64a5e51e39b7d2468c1684754edff6275bb63b663834b581daf33638ea4b322" exitCode=0 Nov 03 19:38:56.378080 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:56.377955 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" event={"ID":"b96c57e6fe4ecb21f4910168c569318a","Type":"ContainerDied","Data":"a64a5e51e39b7d2468c1684754edff6275bb63b663834b581daf33638ea4b322"} Nov 03 19:38:56.384596 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:56.384572 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" event={"ID":"5cf8bc898cbd908afece93ec1ad60d21","Type":"ContainerStarted","Data":"df7ae75ab0d6401a6a7a88804d82484cc0212a4723cdd51e9655983291ce9599"} Nov 03 19:38:56.386410 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:56.386364 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f4x7r" podStartSLOduration=2.220429984 podStartE2EDuration="6.386349169s" totalImagesPullingTime="4.165919185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:56.385208734 +0000 UTC m=+6.781832783" watchObservedRunningTime="2026-11-03 19:38:56.386349169 +0000 UTC m=+6.782973220" Nov 03 19:38:56.776278 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:56.776160 2568 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:38:56.776886 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:56.776861 2568 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:38:56.795583 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:56.795103 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" podStartSLOduration=5.795085895 podStartE2EDuration="5.795085895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:56.415477716 +0000 UTC m=+6.812101768" watchObservedRunningTime="2026-11-03 19:38:56.795085895 +0000 UTC m=+7.191709947" Nov 03 19:38:57.205525 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:57.205454 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:57.205677 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:57.205592 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4t7z" podUID="edae5b46-8158-4dc8-91c9-06ed65126db9" Nov 03 19:38:57.205997 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:57.205917 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:38:57.206098 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:57.206028 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z9svh" podUID="2dcde652-ceea-4a58-9746-eb2087dea702" Nov 03 19:38:57.391978 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:57.391130 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" event={"ID":"b96c57e6fe4ecb21f4910168c569318a","Type":"ContainerStarted","Data":"9083153c1cab1e7b4fa4bb03b8aa6973676185bd2a20dd78ff88bf91d5301648"} Nov 03 19:38:57.396182 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:57.395043 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pvs99" event={"ID":"e90fc007-32bc-43c5-99c6-7dbb3f9753d7","Type":"ContainerStarted","Data":"2cdaf7aac0dec2c013bd71d4990ad97a4c8fb84f8b37c89eeb6256745f52433c"} Nov 03 19:38:57.428098 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:57.428047 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" podStartSLOduration=6.428030301 podStartE2EDuration="6.428030301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:57.407743207 +0000 UTC m=+7.804367258" watchObservedRunningTime="2026-11-03 19:38:57.428030301 +0000 UTC m=+7.824654352" Nov 03 19:38:58.208766 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:58.208025 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:38:58.208766 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:58.208151 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wztqw" podUID="d85c1bad-d10e-4276-9471-0fbe78bb67b3" Nov 03 19:38:58.403390 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:58.402705 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" event={"ID":"04f022dd-949d-40ab-b00c-902e70ef610e","Type":"ContainerStarted","Data":"cacf10185ea7d460a11d52dbd82005c05ae809af36ebbffff48c3d526aafb63e"} Nov 03 19:38:58.403390 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:58.402780 2568 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:38:58.712781 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:58.712023 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs\") pod \"network-metrics-daemon-z9svh\" (UID: \"2dcde652-ceea-4a58-9746-eb2087dea702\") " pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:38:58.712781 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:58.712075 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:58.712781 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:58.712219 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:38:58.712781 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:58.712281 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret podName:edae5b46-8158-4dc8-91c9-06ed65126db9 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:06.712262168 +0000 UTC m=+17.108886213 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret") pod "global-pull-secret-syncer-p4t7z" (UID: "edae5b46-8158-4dc8-91c9-06ed65126db9") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:38:58.712781 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:58.712696 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:38:58.712781 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:58.712744 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs podName:2dcde652-ceea-4a58-9746-eb2087dea702 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:06.712728666 +0000 UTC m=+17.109352710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs") pod "network-metrics-daemon-z9svh" (UID: "2dcde652-ceea-4a58-9746-eb2087dea702") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:38:58.913434 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:58.913356 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22vq\" (UniqueName: \"kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq\") pod \"network-check-target-wztqw\" (UID: \"d85c1bad-d10e-4276-9471-0fbe78bb67b3\") " pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:38:58.913603 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:58.913576 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:38:58.913603 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:58.913597 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:38:58.913709 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:58.913611 2568 projected.go:196] Error preparing data for projected volume kube-api-access-k22vq for pod openshift-network-diagnostics/network-check-target-wztqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:38:58.913709 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:58.913666 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq podName:d85c1bad-d10e-4276-9471-0fbe78bb67b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:06.913648008 +0000 UTC m=+17.310272049 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k22vq" (UniqueName: "kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq") pod "network-check-target-wztqw" (UID: "d85c1bad-d10e-4276-9471-0fbe78bb67b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:38:59.206459 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:59.205818 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:38:59.206459 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:59.205940 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4t7z" podUID="edae5b46-8158-4dc8-91c9-06ed65126db9" Nov 03 19:38:59.206459 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:59.206304 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:38:59.206459 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:38:59.206405 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z9svh" podUID="2dcde652-ceea-4a58-9746-eb2087dea702" Nov 03 19:38:59.235226 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:59.235185 2568 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 19:38:59.409185 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:38:59.409143 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" event={"ID":"04f022dd-949d-40ab-b00c-902e70ef610e","Type":"ContainerStarted","Data":"a36d8a94ea33278008ba7f0de3106865b494adf5b0b898d5c364c6ebecfc9e9e"} Nov 03 19:39:00.207250 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:00.207216 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:00.207383 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:00.207329 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wztqw" podUID="d85c1bad-d10e-4276-9471-0fbe78bb67b3" Nov 03 19:39:00.228772 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:00.228660 2568 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:38:59.235202439Z","UUID":"476d595b-4d27-401b-bdb3-f243b3b4f1d6","Handler":null,"Name":"","Endpoint":""} Nov 03 19:39:00.230759 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:00.230312 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 19:39:00.230759 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:00.230338 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 19:39:00.413759 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:00.413720 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-npg2d" event={"ID":"398cf284-004e-463a-a74e-0193946df8b7","Type":"ContainerStarted","Data":"0e4b1d97b411bac749aa327a3ca95894ac8366ba52525b56577384233868d231"} Nov 03 19:39:00.416576 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:00.416544 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k666m" event={"ID":"29f79e6b-20d5-4ba4-8737-068f0d82b1e6","Type":"ContainerStarted","Data":"f4a90247893dfa36ba7fe505e60710840cb6b2eec19827851d5bc7f8104a6406"} Nov 03 19:39:00.431521 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:00.431480 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pvs99" podStartSLOduration=4.824477057 podStartE2EDuration="10.431469139s" totalImagesPullingTime="5.606992082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:57.428738958 +0000 UTC m=+7.825363009" watchObservedRunningTime="2026-11-03 19:39:00.431469139 +0000 UTC m=+10.828093188" Nov 03 19:39:00.431765 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:00.431727 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-npg2d" podStartSLOduration=2.5825080849999997 podStartE2EDuration="10.43172075s" totalImagesPullingTime="7.849212665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:00.430934351 +0000 UTC m=+10.827558402" watchObservedRunningTime="2026-11-03 19:39:00.43172075 +0000 UTC m=+10.828344800" Nov 03 19:39:00.445473 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:00.445422 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-k666m" podStartSLOduration=2.149580256 podStartE2EDuration="10.445406185s" totalImagesPullingTime="8.295825929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:00.445356694 +0000 UTC m=+10.841980746" watchObservedRunningTime="2026-11-03 19:39:00.445406185 +0000 UTC m=+10.842030235" Nov 03 19:39:01.205901 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:01.205863 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:01.206065 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:01.205912 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:01.206065 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:01.206010 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4t7z" podUID="edae5b46-8158-4dc8-91c9-06ed65126db9" Nov 03 19:39:01.206168 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:01.206115 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z9svh" podUID="2dcde652-ceea-4a58-9746-eb2087dea702" Nov 03 19:39:01.420549 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:01.420501 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rhlgs" event={"ID":"d9e5c9ec-1940-4cb2-8c55-12da74cfc620","Type":"ContainerStarted","Data":"60b83272aea6292a48f801c032fa1032e33c35c31c8dcb5f30f008abbf64ada6"} Nov 03 19:39:01.422707 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:01.422679 2568 generic.go:356] "Generic (PLEG): container finished" podID="36460aa5-b1f0-4be4-b27b-675c6fecffe1" containerID="e6e10c696a9e5ee879102b80645f170a411b1d0283d20677f12313e18e55eb77" exitCode=0 Nov 03 19:39:01.422841 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:01.422766 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6vlx" event={"ID":"36460aa5-b1f0-4be4-b27b-675c6fecffe1","Type":"ContainerDied","Data":"e6e10c696a9e5ee879102b80645f170a411b1d0283d20677f12313e18e55eb77"} Nov 03 19:39:01.434386 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:01.434348 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rhlgs" podStartSLOduration=3.161657165 podStartE2EDuration="11.434336089s" totalImagesPullingTime="8.272678924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:01.434039611 +0000 UTC m=+11.830663666" watchObservedRunningTime="2026-11-03 19:39:01.434336089 +0000 UTC m=+11.830960138" Nov 03 19:39:02.205846 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:02.205665 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:02.205966 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:02.205942 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wztqw" podUID="d85c1bad-d10e-4276-9471-0fbe78bb67b3" Nov 03 19:39:02.428044 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:02.427991 2568 generic.go:356] "Generic (PLEG): container finished" podID="36460aa5-b1f0-4be4-b27b-675c6fecffe1" containerID="3bfcadb20970ec7bb9d2dfcc4a3674a27bdf23ab0189ab81d06c628190219ee9" exitCode=0 Nov 03 19:39:02.428725 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:02.428080 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6vlx" event={"ID":"36460aa5-b1f0-4be4-b27b-675c6fecffe1","Type":"ContainerDied","Data":"3bfcadb20970ec7bb9d2dfcc4a3674a27bdf23ab0189ab81d06c628190219ee9"} Nov 03 19:39:02.429675 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:02.429649 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sks78" event={"ID":"44cfa442-7cca-4e81-af9c-789fb885a12c","Type":"ContainerStarted","Data":"5a18e3846978af072da3a2b0f53584853ec748da884e022c00fbeea283e1d7bb"} Nov 03 19:39:03.206340 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:03.206309 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:03.206512 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:03.206438 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z9svh" podUID="2dcde652-ceea-4a58-9746-eb2087dea702" Nov 03 19:39:03.206512 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:03.206488 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:03.206670 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:03.206642 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4t7z" podUID="edae5b46-8158-4dc8-91c9-06ed65126db9" Nov 03 19:39:03.435053 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:03.435012 2568 generic.go:356] "Generic (PLEG): container finished" podID="36460aa5-b1f0-4be4-b27b-675c6fecffe1" containerID="2cae42d4d47b384bd3c82b237c1af72be88663ad82f48b076bf27d4e9601440d" exitCode=0 Nov 03 19:39:03.435634 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:03.435117 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6vlx" event={"ID":"36460aa5-b1f0-4be4-b27b-675c6fecffe1","Type":"ContainerDied","Data":"2cae42d4d47b384bd3c82b237c1af72be88663ad82f48b076bf27d4e9601440d"} Nov 03 19:39:03.460187 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:03.460116 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sks78" podStartSLOduration=3.329168991 podStartE2EDuration="13.460100083s" totalImagesPullingTime="10.130931092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:02.465856686 +0000 UTC m=+12.862480735" watchObservedRunningTime="2026-11-03 19:39:03.460100083 +0000 UTC m=+13.856724132" Nov 03 19:39:04.206787 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:04.206721 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:04.206952 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:04.206857 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wztqw" podUID="d85c1bad-d10e-4276-9471-0fbe78bb67b3" Nov 03 19:39:05.206518 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:05.206308 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:05.207049 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:05.206350 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:05.207049 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:05.206657 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4t7z" podUID="edae5b46-8158-4dc8-91c9-06ed65126db9" Nov 03 19:39:05.207049 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:05.206736 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z9svh" podUID="2dcde652-ceea-4a58-9746-eb2087dea702" Nov 03 19:39:05.443128 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:05.443100 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" event={"ID":"04f022dd-949d-40ab-b00c-902e70ef610e","Type":"ContainerStarted","Data":"fc2da322cc7462a849b72959088a95391fcc94afbe0d993ae3993ecf4984a49b"} Nov 03 19:39:05.460855 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:05.460755 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" podStartSLOduration=2.703376235 podStartE2EDuration="15.460744096s" totalImagesPullingTime="12.757367861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:05.460642253 +0000 UTC m=+15.857266302" watchObservedRunningTime="2026-11-03 19:39:05.460744096 +0000 UTC m=+15.857368144" Nov 03 19:39:06.205729 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:06.205709 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:06.205813 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:06.205801 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wztqw" podUID="d85c1bad-d10e-4276-9471-0fbe78bb67b3" Nov 03 19:39:06.447704 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:06.447679 2568 generic.go:356] "Generic (PLEG): container finished" podID="36460aa5-b1f0-4be4-b27b-675c6fecffe1" containerID="348f6a0228a555e67e1c61a848d6a87a4d42568f2f75da672741905764ec43a7" exitCode=0 Nov 03 19:39:06.448015 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:06.447752 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6vlx" event={"ID":"36460aa5-b1f0-4be4-b27b-675c6fecffe1","Type":"ContainerDied","Data":"348f6a0228a555e67e1c61a848d6a87a4d42568f2f75da672741905764ec43a7"} Nov 03 19:39:06.450996 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:06.450170 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 19:39:06.450996 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:06.450592 2568 generic.go:356] "Generic (PLEG): container finished" podID="754b8c74-0816-45c8-92f6-95725095e262" containerID="2d5a632d180fdebf9d0372a733e77abeb0f2a16003a3f308ad0bb22eb5edff59" exitCode=1 Nov 03 19:39:06.451108 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:06.451087 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" event={"ID":"754b8c74-0816-45c8-92f6-95725095e262","Type":"ContainerStarted","Data":"19773a15e5a4247524a846c6d467cdb0349dd31bbcc393e26837b1b2bf64a135"} Nov 03 19:39:06.451138 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:06.451133 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" event={"ID":"754b8c74-0816-45c8-92f6-95725095e262","Type":"ContainerDied","Data":"2d5a632d180fdebf9d0372a733e77abeb0f2a16003a3f308ad0bb22eb5edff59"} Nov 03 19:39:06.451179 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:06.451163 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" event={"ID":"754b8c74-0816-45c8-92f6-95725095e262","Type":"ContainerStarted","Data":"8cd145a25de67aa5172f8d63d61d5f7f651a4c105b9dbc18557d047a44b633b5"} Nov 03 19:39:06.774190 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:06.774037 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs\") pod \"network-metrics-daemon-z9svh\" (UID: \"2dcde652-ceea-4a58-9746-eb2087dea702\") " pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:06.774190 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:06.774173 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:39:06.774356 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:06.774197 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:06.774356 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:06.774226 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs podName:2dcde652-ceea-4a58-9746-eb2087dea702 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:22.774211136 +0000 UTC m=+33.170835179 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs") pod "network-metrics-daemon-z9svh" (UID: "2dcde652-ceea-4a58-9746-eb2087dea702") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:39:06.774356 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:06.774275 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:39:06.774356 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:06.774314 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret podName:edae5b46-8158-4dc8-91c9-06ed65126db9 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:22.774303337 +0000 UTC m=+33.170927364 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret") pod "global-pull-secret-syncer-p4t7z" (UID: "edae5b46-8158-4dc8-91c9-06ed65126db9") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:39:06.975856 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:06.975791 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22vq\" (UniqueName: \"kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq\") pod \"network-check-target-wztqw\" (UID: \"d85c1bad-d10e-4276-9471-0fbe78bb67b3\") " pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:06.975977 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:06.975931 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:39:06.975977 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:06.975947 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:39:06.975977 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:06.975955 2568 projected.go:196] Error preparing data for projected volume kube-api-access-k22vq for pod openshift-network-diagnostics/network-check-target-wztqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:39:06.976072 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:06.975998 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq podName:d85c1bad-d10e-4276-9471-0fbe78bb67b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:22.975986196 +0000 UTC m=+33.372610223 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k22vq" (UniqueName: "kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq") pod "network-check-target-wztqw" (UID: "d85c1bad-d10e-4276-9471-0fbe78bb67b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:39:07.205857 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:07.205830 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:07.205857 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:07.205860 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:07.206023 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:07.205927 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z9svh" podUID="2dcde652-ceea-4a58-9746-eb2087dea702" Nov 03 19:39:07.206023 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:07.205976 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4t7z" podUID="edae5b46-8158-4dc8-91c9-06ed65126db9" Nov 03 19:39:07.454931 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:07.454908 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 19:39:07.455320 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:07.455253 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" event={"ID":"754b8c74-0816-45c8-92f6-95725095e262","Type":"ContainerStarted","Data":"564b2aed4eace38534e89f238f67c3ba941575eb5961c6997437e14b85190c6c"} Nov 03 19:39:07.455320 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:07.455314 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" event={"ID":"754b8c74-0816-45c8-92f6-95725095e262","Type":"ContainerStarted","Data":"681c00654073b2a35ef593f70d74237856f322732412addb5acc6151fd2ab1eb"} Nov 03 19:39:07.455435 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:07.455349 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" event={"ID":"754b8c74-0816-45c8-92f6-95725095e262","Type":"ContainerStarted","Data":"c957a5b568507abb96f7f2aca214c27d4e0a2e3e7e110960415d7dfd5ca4caec"} Nov 03 19:39:07.457850 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:07.457829 2568 generic.go:356] "Generic (PLEG): container finished" podID="36460aa5-b1f0-4be4-b27b-675c6fecffe1" containerID="4dd5fabd891fa76438ef2de301c3e15d52ae5e95f6c7b9ca09bf0f1995e8b47f" exitCode=0 Nov 03 19:39:07.457949 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:07.457856 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6vlx" event={"ID":"36460aa5-b1f0-4be4-b27b-675c6fecffe1","Type":"ContainerDied","Data":"4dd5fabd891fa76438ef2de301c3e15d52ae5e95f6c7b9ca09bf0f1995e8b47f"} Nov 03 19:39:08.205934 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:08.205903 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:08.206205 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:08.205994 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wztqw" podUID="d85c1bad-d10e-4276-9471-0fbe78bb67b3" Nov 03 19:39:08.462795 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:08.462724 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6vlx" event={"ID":"36460aa5-b1f0-4be4-b27b-675c6fecffe1","Type":"ContainerStarted","Data":"729a664c7a5861ad213f487bebd8648fe711a84a354e431061cfa1ed33b7460d"} Nov 03 19:39:08.483843 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:08.483807 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w6vlx" podStartSLOduration=7.20762758 podStartE2EDuration="18.483795574s" totalImagesPullingTime="11.276167994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:08.483402017 +0000 UTC m=+18.880026079" watchObservedRunningTime="2026-11-03 19:39:08.483795574 +0000 UTC m=+18.880419621" Nov 03 19:39:09.206714 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:09.206682 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:09.207070 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:09.207048 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z9svh" podUID="2dcde652-ceea-4a58-9746-eb2087dea702" Nov 03 19:39:09.207121 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:09.207069 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:09.207160 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:09.207139 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4t7z" podUID="edae5b46-8158-4dc8-91c9-06ed65126db9" Nov 03 19:39:09.467682 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:09.467620 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 19:39:09.468142 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:09.467956 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" event={"ID":"754b8c74-0816-45c8-92f6-95725095e262","Type":"ContainerStarted","Data":"5b0f42705290d643e423069305d4ce49a3f30f2c1c0e8fc125facdf1c285ac0a"} Nov 03 19:39:10.206236 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:10.206207 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:10.206395 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:10.206287 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wztqw" podUID="d85c1bad-d10e-4276-9471-0fbe78bb67b3" Nov 03 19:39:11.205902 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:11.205874 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:11.206372 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:11.205975 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z9svh" podUID="2dcde652-ceea-4a58-9746-eb2087dea702" Nov 03 19:39:11.206372 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:11.206046 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:11.206372 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:11.206127 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4t7z" podUID="edae5b46-8158-4dc8-91c9-06ed65126db9" Nov 03 19:39:11.474897 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:11.474836 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 19:39:11.475181 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:11.475160 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" event={"ID":"754b8c74-0816-45c8-92f6-95725095e262","Type":"ContainerStarted","Data":"fcc7cd3a413e44add7877e42b21717c19548780bc81a336eee1bbe197b025653"} Nov 03 19:39:11.475518 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:11.475493 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:11.475631 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:11.475582 2568 scope.go:122] "RemoveContainer" containerID="2d5a632d180fdebf9d0372a733e77abeb0f2a16003a3f308ad0bb22eb5edff59" Nov 03 19:39:11.492104 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:11.492084 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:12.206510 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:12.206345 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:12.206887 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:12.206611 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wztqw" podUID="d85c1bad-d10e-4276-9471-0fbe78bb67b3" Nov 03 19:39:12.460365 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:12.460301 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p4t7z"] Nov 03 19:39:12.460503 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:12.460445 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:12.460606 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:12.460584 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4t7z" podUID="edae5b46-8158-4dc8-91c9-06ed65126db9" Nov 03 19:39:12.463215 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:12.463187 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z9svh"] Nov 03 19:39:12.463353 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:12.463303 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:12.463437 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:12.463410 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z9svh" podUID="2dcde652-ceea-4a58-9746-eb2087dea702" Nov 03 19:39:12.463886 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:12.463866 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wztqw"] Nov 03 19:39:12.481606 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:12.481580 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 19:39:12.481960 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:12.481942 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:12.482045 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:12.481983 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" event={"ID":"754b8c74-0816-45c8-92f6-95725095e262","Type":"ContainerStarted","Data":"444cb1a21125cc36e8d8cdeb079ccdaa7e9951b1c267b49fc3d2e7bf6feeb045"} Nov 03 19:39:12.482045 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:12.482016 2568 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:39:12.482139 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:12.482015 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wztqw" podUID="d85c1bad-d10e-4276-9471-0fbe78bb67b3" Nov 03 19:39:12.482260 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:12.482237 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:12.497027 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:12.497008 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:12.509885 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:12.509849 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" podStartSLOduration=7.860430407 podStartE2EDuration="22.509838663s" totalImagesPullingTime="14.649408256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:12.508249759 +0000 UTC m=+22.904873811" watchObservedRunningTime="2026-11-03 19:39:12.509838663 +0000 UTC m=+22.906462704" Nov 03 19:39:13.484087 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:13.484060 2568 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:39:14.205895 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:14.205869 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:14.206200 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:14.205901 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:14.206200 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:14.205991 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4t7z" podUID="edae5b46-8158-4dc8-91c9-06ed65126db9" Nov 03 19:39:14.206200 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:14.206018 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:14.206200 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:14.206077 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wztqw" podUID="d85c1bad-d10e-4276-9471-0fbe78bb67b3" Nov 03 19:39:14.206200 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:14.206155 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z9svh" podUID="2dcde652-ceea-4a58-9746-eb2087dea702" Nov 03 19:39:14.485975 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:14.485896 2568 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:39:16.206363 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:16.206328 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:16.206775 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:16.206331 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:16.206775 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:16.206469 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z9svh" podUID="2dcde652-ceea-4a58-9746-eb2087dea702" Nov 03 19:39:16.206775 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:16.206337 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:16.206775 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:16.206512 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4t7z" podUID="edae5b46-8158-4dc8-91c9-06ed65126db9" Nov 03 19:39:16.206775 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:16.206592 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wztqw" podUID="d85c1bad-d10e-4276-9471-0fbe78bb67b3" Nov 03 19:39:17.954408 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:17.954383 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeReady" Nov 03 19:39:17.954800 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:17.954513 2568 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 19:39:17.987451 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:17.987416 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-knv7j"] Nov 03 19:39:17.990107 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:17.990084 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-knv7j" Nov 03 19:39:17.990447 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:17.990427 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-x9cw5"] Nov 03 19:39:17.993609 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:17.993578 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-d5qxk"] Nov 03 19:39:17.993725 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:17.993680 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-x9cw5" Nov 03 19:39:17.996360 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:17.996344 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b48f6466f-rvj5f"] Nov 03 19:39:17.996483 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:17.996465 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-d5qxk" Nov 03 19:39:17.999203 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:17.999186 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.001833 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.001815 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-855cd4cff-prjqj"] Nov 03 19:39:18.004220 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.004201 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 19:39:18.004374 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.004355 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 19:39:18.004498 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.004478 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-rs8wk"] Nov 03 19:39:18.004711 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.004649 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:18.007394 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.007378 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8"] Nov 03 19:39:18.007499 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.007487 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.010276 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.010255 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-txtq2"] Nov 03 19:39:18.010452 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.010370 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:18.013552 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.013521 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p"] Nov 03 19:39:18.013660 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.013641 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-txtq2" Nov 03 19:39:18.016365 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.016344 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-68j2z"] Nov 03 19:39:18.016478 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.016461 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:18.019783 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.019765 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85444c64c6-qk7p8"] Nov 03 19:39:18.020055 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.020037 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:18.023177 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.023159 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.029421 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.029404 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-f9vq4"] Nov 03 19:39:18.032382 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.032295 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f9vq4" Nov 03 19:39:18.051678 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.051653 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 19:39:18.051678 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.051673 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 19:39:18.052722 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.052706 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 19:39:18.052814 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.052798 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x24rb\"" Nov 03 19:39:18.052857 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.052812 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 19:39:18.053496 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.053478 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-sk4zs"] Nov 03 19:39:18.058348 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.056894 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mwk25\"" Nov 03 19:39:18.058348 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.056953 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 19:39:18.058348 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.056983 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 19:39:18.058348 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.057099 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-st4g7\"" Nov 03 19:39:18.058348 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.058311 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck"] Nov 03 19:39:18.058793 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.058771 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:18.061872 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.061851 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4"] Nov 03 19:39:18.061987 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.061970 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:18.064846 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.064829 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-d5qxk"] Nov 03 19:39:18.064923 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.064855 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-knv7j"] Nov 03 19:39:18.064923 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.064871 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b48f6466f-rvj5f"] Nov 03 19:39:18.064923 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.064884 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8"] Nov 03 19:39:18.064923 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.064897 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-855cd4cff-prjqj"] Nov 03 19:39:18.065129 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.064957 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:18.068688 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.068666 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-txtq2"] Nov 03 19:39:18.073812 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.072520 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85444c64c6-qk7p8"] Nov 03 19:39:18.074603 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.074589 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zdf6j"] Nov 03 19:39:18.078972 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.078948 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-f9vq4"] Nov 03 19:39:18.079077 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.078999 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ggbpk"] Nov 03 19:39:18.079483 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.079445 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:18.087924 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.087710 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-x9cw5"] Nov 03 19:39:18.087924 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.087755 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-sk4zs"] Nov 03 19:39:18.087924 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.087873 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ggbpk" Nov 03 19:39:18.088145 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.088054 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p"] Nov 03 19:39:18.088145 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.088138 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck"] Nov 03 19:39:18.088247 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.088171 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-rs8wk"] Nov 03 19:39:18.089609 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.089590 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-68j2z"] Nov 03 19:39:18.092682 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.092440 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4"] Nov 03 19:39:18.094644 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.094622 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zdf6j"] Nov 03 19:39:18.098206 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.098110 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ggbpk"] Nov 03 19:39:18.104349 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104321 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 19:39:18.104442 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104350 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 19:39:18.104442 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104408 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 19:39:18.104564 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104444 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 19:39:18.104564 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104461 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 19:39:18.104564 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104524 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 19:39:18.104718 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104635 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 19:39:18.104718 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104679 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 19:39:18.104718 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104686 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 19:39:18.104854 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104740 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 19:39:18.104854 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104758 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 19:39:18.104854 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104771 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 19:39:18.104854 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104779 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:39:18.104854 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104816 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:39:18.104854 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104827 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 19:39:18.104854 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104820 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 19:39:18.104854 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104836 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 19:39:18.105215 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104871 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 19:39:18.105215 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104872 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:39:18.105215 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104909 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 19:39:18.105215 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104984 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 19:39:18.105215 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.104995 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 19:39:18.105215 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.105013 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 19:39:18.105215 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.105023 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 19:39:18.105215 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.105036 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:39:18.105215 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.105042 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 19:39:18.105215 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.105044 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:39:18.105215 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.105064 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 19:39:18.108024 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.108009 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 19:39:18.111098 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.111075 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 19:39:18.111738 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.111721 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 19:39:18.119936 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.119918 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 19:39:18.120284 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120266 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 19:39:18.120353 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120331 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 19:39:18.120434 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120415 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 19:39:18.120434 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120426 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-p5v54\"" Nov 03 19:39:18.120525 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120420 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fx66q\"" Nov 03 19:39:18.120525 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120439 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 19:39:18.120525 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120449 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6gncv\"" Nov 03 19:39:18.120525 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120481 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 19:39:18.120525 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120491 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 19:39:18.120525 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120517 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 19:39:18.120782 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120583 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 19:39:18.120782 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120664 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 19:39:18.120782 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120699 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tlzjx\"" Nov 03 19:39:18.120782 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120757 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gchmt\"" Nov 03 19:39:18.120782 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120758 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-sstjk\"" Nov 03 19:39:18.121012 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120807 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-869qf\"" Nov 03 19:39:18.121012 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120840 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mvxmh\"" Nov 03 19:39:18.121012 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120898 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q82hb\"" Nov 03 19:39:18.121012 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.120951 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-fk8jl\"" Nov 03 19:39:18.151945 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.151911 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-trusted-ca\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.151945 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.151946 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-stats-auth\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:18.152139 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.151972 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-installation-pull-secrets\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.152139 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.151995 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73f2218a-88f7-4f26-98b3-2d6403bff02d-serving-cert\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:18.152139 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152099 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4w67\" (UniqueName: \"kubernetes.io/projected/69c72eb2-3839-4483-82e8-b7d0bf82c57e-kube-api-access-f4w67\") pod \"service-ca-operator-544d749c8c-bqmb8\" (UID: \"69c72eb2-3839-4483-82e8-b7d0bf82c57e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:18.152139 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152132 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-image-registry-private-configuration\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.152276 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152152 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25n72\" (UniqueName: \"kubernetes.io/projected/51c48ea1-6b42-4ec0-ae95-bf81b913e901-kube-api-access-25n72\") pod \"kas-connection-checker-685895c7c9-d5qxk\" (UID: \"51c48ea1-6b42-4ec0-ae95-bf81b913e901\") " pod="kube-system/kas-connection-checker-685895c7c9-d5qxk" Nov 03 19:39:18.152276 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152176 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpk6g\" (UniqueName: \"kubernetes.io/projected/1b0776e1-a654-4e61-ba94-6d81ddfc8877-kube-api-access-fpk6g\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:18.152276 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152206 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7qbv\" (UniqueName: \"kubernetes.io/projected/09d5c2e9-13cd-4859-abbf-218d18877c5a-kube-api-access-s7qbv\") pod \"network-check-source-6f78d59f99-f9vq4\" (UID: \"09d5c2e9-13cd-4859-abbf-218d18877c5a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f9vq4" Nov 03 19:39:18.152276 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152237 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.152276 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152258 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-bound-sa-token\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.152438 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152307 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-certificates\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.152438 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152336 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2266dada-9604-44f9-b488-511006cee7b2-serving-cert\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.152438 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152359 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-ca-trust-extracted\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.152438 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152394 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:18.152438 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152414 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5101a457-b490-4538-a9f6-288014fa9264-tmp-dir\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:18.152639 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152447 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74a9a94d-e73f-4942-bab6-efbfcb728df2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p\" (UID: \"74a9a94d-e73f-4942-bab6-efbfcb728df2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:18.152639 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152468 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:18.152639 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152563 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5101a457-b490-4538-a9f6-288014fa9264-config-volume\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:18.152639 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152600 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qx5n\" (UniqueName: \"kubernetes.io/projected/ca34d416-e238-48cb-a507-953f78fead22-kube-api-access-4qx5n\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:18.152769 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152645 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74a9a94d-e73f-4942-bab6-efbfcb728df2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p\" (UID: \"74a9a94d-e73f-4942-bab6-efbfcb728df2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:18.152769 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152662 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvzw\" (UniqueName: \"kubernetes.io/projected/74a9a94d-e73f-4942-bab6-efbfcb728df2-kube-api-access-mmvzw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p\" (UID: \"74a9a94d-e73f-4942-bab6-efbfcb728df2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:18.152769 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152698 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-bound-sa-token\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.152769 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152722 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-image-registry-private-configuration\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.152769 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152749 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-installation-pull-secrets\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.152957 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152766 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2266dada-9604-44f9-b488-511006cee7b2-service-ca-bundle\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.152957 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152817 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73f2218a-88f7-4f26-98b3-2d6403bff02d-config\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:18.152957 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152851 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgc9h\" (UniqueName: \"kubernetes.io/projected/5101a457-b490-4538-a9f6-288014fa9264-kube-api-access-rgc9h\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:18.152957 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152871 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hcnk\" (UniqueName: \"kubernetes.io/projected/2266dada-9604-44f9-b488-511006cee7b2-kube-api-access-2hcnk\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.152957 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152953 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69c72eb2-3839-4483-82e8-b7d0bf82c57e-serving-cert\") pod \"service-ca-operator-544d749c8c-bqmb8\" (UID: \"69c72eb2-3839-4483-82e8-b7d0bf82c57e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:18.153146 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152971 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fvg7\" (UniqueName: \"kubernetes.io/projected/257528a3-a0f4-4a07-bba0-67561f8ee205-kube-api-access-9fvg7\") pod \"kas-connection-checker-685895c7c9-x9cw5\" (UID: \"257528a3-a0f4-4a07-bba0-67561f8ee205\") " pod="kube-system/kas-connection-checker-685895c7c9-x9cw5" Nov 03 19:39:18.153146 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.152992 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-registry-certificates\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.153146 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153008 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.153146 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153048 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:18.153146 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153081 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4hfr\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-kube-api-access-g4hfr\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.153358 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153143 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2266dada-9604-44f9-b488-511006cee7b2-snapshots\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.153358 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153170 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-trusted-ca\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.153358 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153208 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dph2k\" (UniqueName: \"kubernetes.io/projected/d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041-kube-api-access-dph2k\") pod \"kas-connection-checker-685895c7c9-knv7j\" (UID: \"d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041\") " pod="kube-system/kas-connection-checker-685895c7c9-knv7j" Nov 03 19:39:18.153358 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153224 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2266dada-9604-44f9-b488-511006cee7b2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.153358 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153286 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5j6b\" (UniqueName: \"kubernetes.io/projected/0fbfc914-8c97-4770-9ef6-6e553c4dcc58-kube-api-access-r5j6b\") pod \"volume-data-source-validator-58c9f5f46d-txtq2\" (UID: \"0fbfc914-8c97-4770-9ef6-6e553c4dcc58\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-txtq2" Nov 03 19:39:18.153358 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153321 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-default-certificate\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:18.153358 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153350 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:18.153637 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153376 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:18.153637 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153439 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6bab2991-93a7-44c8-9071-e3c971b79885-ca-trust-extracted\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.153637 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153457 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c67h2\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-kube-api-access-c67h2\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.153637 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153514 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2266dada-9604-44f9-b488-511006cee7b2-tmp\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.153637 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153571 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73f2218a-88f7-4f26-98b3-2d6403bff02d-trusted-ca\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:18.153637 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153601 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sk4zs\" (UID: \"b2503201-686a-4cf9-b5a1-977c87e6b717\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:18.153850 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153645 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/69c72eb2-3839-4483-82e8-b7d0bf82c57e-config\") pod \"service-ca-operator-544d749c8c-bqmb8\" (UID: \"69c72eb2-3839-4483-82e8-b7d0bf82c57e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:18.153850 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153673 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:18.153850 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153697 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pp9v\" (UniqueName: \"kubernetes.io/projected/73f2218a-88f7-4f26-98b3-2d6403bff02d-kube-api-access-9pp9v\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:18.153850 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153727 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2503201-686a-4cf9-b5a1-977c87e6b717-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sk4zs\" (UID: \"b2503201-686a-4cf9-b5a1-977c87e6b717\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:18.153850 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153754 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ca34d416-e238-48cb-a507-953f78fead22-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:18.153850 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.153795 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gk22\" (UniqueName: \"kubernetes.io/projected/9924fd9e-f7b6-4036-8584-22caa1de39b3-kube-api-access-7gk22\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:18.205930 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.205901 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:18.205930 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.205922 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:18.206141 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.205909 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:18.220343 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.220317 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-drbgz\"" Nov 03 19:39:18.220343 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.220339 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5rk8\"" Nov 03 19:39:18.220578 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.220564 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zcfm6\"" Nov 03 19:39:18.223625 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.220642 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 19:39:18.223625 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.220698 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 19:39:18.255130 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255101 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-installation-pull-secrets\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.255130 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255133 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73f2218a-88f7-4f26-98b3-2d6403bff02d-serving-cert\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:18.255322 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255157 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4w67\" (UniqueName: \"kubernetes.io/projected/69c72eb2-3839-4483-82e8-b7d0bf82c57e-kube-api-access-f4w67\") pod \"service-ca-operator-544d749c8c-bqmb8\" (UID: \"69c72eb2-3839-4483-82e8-b7d0bf82c57e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:18.255322 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255179 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-image-registry-private-configuration\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.255322 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255194 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25n72\" (UniqueName: \"kubernetes.io/projected/51c48ea1-6b42-4ec0-ae95-bf81b913e901-kube-api-access-25n72\") pod \"kas-connection-checker-685895c7c9-d5qxk\" (UID: \"51c48ea1-6b42-4ec0-ae95-bf81b913e901\") " pod="kube-system/kas-connection-checker-685895c7c9-d5qxk" Nov 03 19:39:18.255480 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpk6g\" (UniqueName: \"kubernetes.io/projected/1b0776e1-a654-4e61-ba94-6d81ddfc8877-kube-api-access-fpk6g\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:18.255480 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255402 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7qbv\" (UniqueName: \"kubernetes.io/projected/09d5c2e9-13cd-4859-abbf-218d18877c5a-kube-api-access-s7qbv\") pod \"network-check-source-6f78d59f99-f9vq4\" (UID: \"09d5c2e9-13cd-4859-abbf-218d18877c5a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f9vq4" Nov 03 19:39:18.255480 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255429 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.255480 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255457 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-bound-sa-token\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.255682 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255485 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-certificates\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.255682 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255509 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2266dada-9604-44f9-b488-511006cee7b2-serving-cert\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.255682 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-ca-trust-extracted\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.255682 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255574 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:18.255682 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255602 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5101a457-b490-4538-a9f6-288014fa9264-tmp-dir\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:18.255682 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255628 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74a9a94d-e73f-4942-bab6-efbfcb728df2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p\" (UID: \"74a9a94d-e73f-4942-bab6-efbfcb728df2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:18.255682 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:18.255985 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.255687 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:39:18.255985 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.255707 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85444c64c6-qk7p8: secret "image-registry-tls" not found Nov 03 19:39:18.255985 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255690 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4734a1de-ee4e-42ab-b925-1a85f65b7d24-cert\") pod \"ingress-canary-ggbpk\" (UID: \"4734a1de-ee4e-42ab-b925-1a85f65b7d24\") " pod="openshift-ingress-canary/ingress-canary-ggbpk" Nov 03 19:39:18.255985 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255742 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hdcb\" (UniqueName: \"kubernetes.io/projected/4734a1de-ee4e-42ab-b925-1a85f65b7d24-kube-api-access-9hdcb\") pod \"ingress-canary-ggbpk\" (UID: \"4734a1de-ee4e-42ab-b925-1a85f65b7d24\") " pod="openshift-ingress-canary/ingress-canary-ggbpk" Nov 03 19:39:18.255985 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.255763 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls podName:ac78f686-0d5a-4ff1-8a04-101e5794b7d7 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:18.755747061 +0000 UTC m=+29.152371117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls") pod "image-registry-85444c64c6-qk7p8" (UID: "ac78f686-0d5a-4ff1-8a04-101e5794b7d7") : secret "image-registry-tls" not found Nov 03 19:39:18.255985 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.255794 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:39:18.255985 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.255845 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:39:18.255985 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.255880 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls podName:1b0776e1-a654-4e61-ba94-6d81ddfc8877 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:18.755863699 +0000 UTC m=+29.152487729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-szqck" (UID: "1b0776e1-a654-4e61-ba94-6d81ddfc8877") : secret "samples-operator-tls" not found Nov 03 19:39:18.255985 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255913 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5101a457-b490-4538-a9f6-288014fa9264-config-volume\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:18.255985 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.255929 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs podName:9924fd9e-f7b6-4036-8584-22caa1de39b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:18.755912256 +0000 UTC m=+29.152536290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs") pod "router-default-855cd4cff-prjqj" (UID: "9924fd9e-f7b6-4036-8584-22caa1de39b3") : secret "router-metrics-certs-default" not found Nov 03 19:39:18.256391 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.255988 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qx5n\" (UniqueName: \"kubernetes.io/projected/ca34d416-e238-48cb-a507-953f78fead22-kube-api-access-4qx5n\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:18.256391 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256020 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74a9a94d-e73f-4942-bab6-efbfcb728df2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p\" (UID: \"74a9a94d-e73f-4942-bab6-efbfcb728df2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:18.256391 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256045 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvzw\" (UniqueName: \"kubernetes.io/projected/74a9a94d-e73f-4942-bab6-efbfcb728df2-kube-api-access-mmvzw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p\" (UID: \"74a9a94d-e73f-4942-bab6-efbfcb728df2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:18.256391 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256073 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-bound-sa-token\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.256391 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256097 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-image-registry-private-configuration\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.256391 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256125 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-installation-pull-secrets\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.256391 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256152 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2266dada-9604-44f9-b488-511006cee7b2-service-ca-bundle\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.256391 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256181 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73f2218a-88f7-4f26-98b3-2d6403bff02d-config\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:18.256391 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256208 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgc9h\" (UniqueName: \"kubernetes.io/projected/5101a457-b490-4538-a9f6-288014fa9264-kube-api-access-rgc9h\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:18.256391 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256233 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hcnk\" (UniqueName: \"kubernetes.io/projected/2266dada-9604-44f9-b488-511006cee7b2-kube-api-access-2hcnk\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.256391 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256256 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-ca-trust-extracted\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.256391 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256263 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69c72eb2-3839-4483-82e8-b7d0bf82c57e-serving-cert\") pod \"service-ca-operator-544d749c8c-bqmb8\" (UID: \"69c72eb2-3839-4483-82e8-b7d0bf82c57e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:18.256391 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256289 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fvg7\" (UniqueName: \"kubernetes.io/projected/257528a3-a0f4-4a07-bba0-67561f8ee205-kube-api-access-9fvg7\") pod \"kas-connection-checker-685895c7c9-x9cw5\" (UID: \"257528a3-a0f4-4a07-bba0-67561f8ee205\") " pod="kube-system/kas-connection-checker-685895c7c9-x9cw5" Nov 03 19:39:18.256391 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256330 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-registry-certificates\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.256391 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256361 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.257028 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256370 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-certificates\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.257028 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256383 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:18.257028 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256411 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4hfr\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-kube-api-access-g4hfr\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.257028 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256450 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5101a457-b490-4538-a9f6-288014fa9264-config-volume\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:18.257028 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256623 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2266dada-9604-44f9-b488-511006cee7b2-snapshots\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.257028 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256652 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-trusted-ca\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.257028 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256683 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dph2k\" (UniqueName: \"kubernetes.io/projected/d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041-kube-api-access-dph2k\") pod \"kas-connection-checker-685895c7c9-knv7j\" (UID: \"d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041\") " pod="kube-system/kas-connection-checker-685895c7c9-knv7j" Nov 03 19:39:18.257028 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256706 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2266dada-9604-44f9-b488-511006cee7b2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.257028 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256743 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5j6b\" (UniqueName: \"kubernetes.io/projected/0fbfc914-8c97-4770-9ef6-6e553c4dcc58-kube-api-access-r5j6b\") pod \"volume-data-source-validator-58c9f5f46d-txtq2\" (UID: \"0fbfc914-8c97-4770-9ef6-6e553c4dcc58\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-txtq2" Nov 03 19:39:18.257028 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256764 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-default-certificate\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:18.257028 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256785 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:18.257028 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256809 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:18.257028 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256846 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6bab2991-93a7-44c8-9071-e3c971b79885-ca-trust-extracted\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.257028 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256867 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c67h2\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-kube-api-access-c67h2\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.257028 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256887 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2266dada-9604-44f9-b488-511006cee7b2-tmp\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.257028 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256910 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73f2218a-88f7-4f26-98b3-2d6403bff02d-trusted-ca\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:18.257753 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256936 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sk4zs\" (UID: \"b2503201-686a-4cf9-b5a1-977c87e6b717\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:18.257753 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256961 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/69c72eb2-3839-4483-82e8-b7d0bf82c57e-config\") pod \"service-ca-operator-544d749c8c-bqmb8\" (UID: \"69c72eb2-3839-4483-82e8-b7d0bf82c57e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:18.257753 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.256985 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:18.257753 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.257014 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pp9v\" (UniqueName: \"kubernetes.io/projected/73f2218a-88f7-4f26-98b3-2d6403bff02d-kube-api-access-9pp9v\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:18.257753 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.257038 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2503201-686a-4cf9-b5a1-977c87e6b717-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sk4zs\" (UID: \"b2503201-686a-4cf9-b5a1-977c87e6b717\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:18.257753 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.257067 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ca34d416-e238-48cb-a507-953f78fead22-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:18.257753 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.257117 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gk22\" (UniqueName: \"kubernetes.io/projected/9924fd9e-f7b6-4036-8584-22caa1de39b3-kube-api-access-7gk22\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:18.257753 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.257153 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-trusted-ca\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.257753 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.257175 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-stats-auth\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:18.258159 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.257997 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2266dada-9604-44f9-b488-511006cee7b2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.258923 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.258899 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-registry-certificates\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.259062 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.258979 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-trusted-ca\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.259134 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.259032 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:39:18.259134 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.259075 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b48f6466f-rvj5f: secret "image-registry-tls" not found Nov 03 19:39:18.259234 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.259187 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls podName:6bab2991-93a7-44c8-9071-e3c971b79885 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:18.759155831 +0000 UTC m=+29.155779859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls") pod "image-registry-b48f6466f-rvj5f" (UID: "6bab2991-93a7-44c8-9071-e3c971b79885") : secret "image-registry-tls" not found Nov 03 19:39:18.259234 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.259213 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5101a457-b490-4538-a9f6-288014fa9264-tmp-dir\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:18.260116 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.260091 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73f2218a-88f7-4f26-98b3-2d6403bff02d-trusted-ca\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:18.260212 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.260099 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2266dada-9604-44f9-b488-511006cee7b2-snapshots\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.260292 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.260275 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:39:18.260352 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.260339 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert podName:b2503201-686a-4cf9-b5a1-977c87e6b717 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:18.760324931 +0000 UTC m=+29.156948977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sk4zs" (UID: "b2503201-686a-4cf9-b5a1-977c87e6b717") : secret "networking-console-plugin-cert" not found Nov 03 19:39:18.260420 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.260365 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2266dada-9604-44f9-b488-511006cee7b2-serving-cert\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.260420 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.260375 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74a9a94d-e73f-4942-bab6-efbfcb728df2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p\" (UID: \"74a9a94d-e73f-4942-bab6-efbfcb728df2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:18.260631 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.260609 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73f2218a-88f7-4f26-98b3-2d6403bff02d-config\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:18.260859 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.260839 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2266dada-9604-44f9-b488-511006cee7b2-service-ca-bundle\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.260913 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.260898 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/69c72eb2-3839-4483-82e8-b7d0bf82c57e-config\") pod \"service-ca-operator-544d749c8c-bqmb8\" (UID: \"69c72eb2-3839-4483-82e8-b7d0bf82c57e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:18.260965 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.260919 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:39:18.260965 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.260913 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6bab2991-93a7-44c8-9071-e3c971b79885-ca-trust-extracted\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.261055 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.260975 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls podName:ca34d416-e238-48cb-a507-953f78fead22 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:18.760961818 +0000 UTC m=+29.157585865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zx2m4" (UID: "ca34d416-e238-48cb-a507-953f78fead22") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:39:18.261055 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.261051 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle podName:9924fd9e-f7b6-4036-8584-22caa1de39b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:18.761040466 +0000 UTC m=+29.157664499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle") pod "router-default-855cd4cff-prjqj" (UID: "9924fd9e-f7b6-4036-8584-22caa1de39b3") : configmap references non-existent config key: service-ca.crt Nov 03 19:39:18.261654 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.261626 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:18.261738 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.261634 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ca34d416-e238-48cb-a507-953f78fead22-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:18.261738 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.261682 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2503201-686a-4cf9-b5a1-977c87e6b717-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sk4zs\" (UID: \"b2503201-686a-4cf9-b5a1-977c87e6b717\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:18.261738 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.261698 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:39:18.262487 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.262461 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-installation-pull-secrets\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.262487 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.262484 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls podName:5101a457-b490-4538-a9f6-288014fa9264 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:18.76246609 +0000 UTC m=+29.159090122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls") pod "dns-default-zdf6j" (UID: "5101a457-b490-4538-a9f6-288014fa9264") : secret "dns-default-metrics-tls" not found Nov 03 19:39:18.262737 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.262466 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-stats-auth\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:18.262990 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.262861 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73f2218a-88f7-4f26-98b3-2d6403bff02d-serving-cert\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:18.263065 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.263046 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-image-registry-private-configuration\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.263336 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.263316 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74a9a94d-e73f-4942-bab6-efbfcb728df2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p\" (UID: \"74a9a94d-e73f-4942-bab6-efbfcb728df2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:18.263414 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.263314 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2266dada-9604-44f9-b488-511006cee7b2-tmp\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.263910 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.263881 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-trusted-ca\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.263992 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.263947 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69c72eb2-3839-4483-82e8-b7d0bf82c57e-serving-cert\") pod \"service-ca-operator-544d749c8c-bqmb8\" (UID: \"69c72eb2-3839-4483-82e8-b7d0bf82c57e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:18.264221 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.264201 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-installation-pull-secrets\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.264327 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.264247 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-image-registry-private-configuration\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.264520 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.264500 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-default-certificate\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:18.268807 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.268763 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4w67\" (UniqueName: \"kubernetes.io/projected/69c72eb2-3839-4483-82e8-b7d0bf82c57e-kube-api-access-f4w67\") pod \"service-ca-operator-544d749c8c-bqmb8\" (UID: \"69c72eb2-3839-4483-82e8-b7d0bf82c57e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:18.270256 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.270133 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7qbv\" (UniqueName: \"kubernetes.io/projected/09d5c2e9-13cd-4859-abbf-218d18877c5a-kube-api-access-s7qbv\") pod \"network-check-source-6f78d59f99-f9vq4\" (UID: \"09d5c2e9-13cd-4859-abbf-218d18877c5a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f9vq4" Nov 03 19:39:18.270789 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.270765 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-bound-sa-token\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.271208 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.271184 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25n72\" (UniqueName: \"kubernetes.io/projected/51c48ea1-6b42-4ec0-ae95-bf81b913e901-kube-api-access-25n72\") pod \"kas-connection-checker-685895c7c9-d5qxk\" (UID: \"51c48ea1-6b42-4ec0-ae95-bf81b913e901\") " pod="kube-system/kas-connection-checker-685895c7c9-d5qxk" Nov 03 19:39:18.272149 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.272122 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-bound-sa-token\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.272409 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.272386 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fvg7\" (UniqueName: \"kubernetes.io/projected/257528a3-a0f4-4a07-bba0-67561f8ee205-kube-api-access-9fvg7\") pod \"kas-connection-checker-685895c7c9-x9cw5\" (UID: \"257528a3-a0f4-4a07-bba0-67561f8ee205\") " pod="kube-system/kas-connection-checker-685895c7c9-x9cw5" Nov 03 19:39:18.274151 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.274125 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgc9h\" (UniqueName: \"kubernetes.io/projected/5101a457-b490-4538-a9f6-288014fa9264-kube-api-access-rgc9h\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:18.274238 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.274147 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dph2k\" (UniqueName: \"kubernetes.io/projected/d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041-kube-api-access-dph2k\") pod \"kas-connection-checker-685895c7c9-knv7j\" (UID: \"d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041\") " pod="kube-system/kas-connection-checker-685895c7c9-knv7j" Nov 03 19:39:18.274348 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.274320 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qx5n\" (UniqueName: \"kubernetes.io/projected/ca34d416-e238-48cb-a507-953f78fead22-kube-api-access-4qx5n\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:18.274766 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.274738 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hcnk\" (UniqueName: \"kubernetes.io/projected/2266dada-9604-44f9-b488-511006cee7b2-kube-api-access-2hcnk\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.274856 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.274775 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c67h2\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-kube-api-access-c67h2\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.274926 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.274908 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gk22\" (UniqueName: \"kubernetes.io/projected/9924fd9e-f7b6-4036-8584-22caa1de39b3-kube-api-access-7gk22\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:18.274974 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.274942 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5j6b\" (UniqueName: \"kubernetes.io/projected/0fbfc914-8c97-4770-9ef6-6e553c4dcc58-kube-api-access-r5j6b\") pod \"volume-data-source-validator-58c9f5f46d-txtq2\" (UID: \"0fbfc914-8c97-4770-9ef6-6e553c4dcc58\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-txtq2" Nov 03 19:39:18.275316 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.275300 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpk6g\" (UniqueName: \"kubernetes.io/projected/1b0776e1-a654-4e61-ba94-6d81ddfc8877-kube-api-access-fpk6g\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:18.275596 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.275572 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvzw\" (UniqueName: \"kubernetes.io/projected/74a9a94d-e73f-4942-bab6-efbfcb728df2-kube-api-access-mmvzw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p\" (UID: \"74a9a94d-e73f-4942-bab6-efbfcb728df2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:18.275841 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.275822 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4hfr\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-kube-api-access-g4hfr\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.275894 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.275828 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pp9v\" (UniqueName: \"kubernetes.io/projected/73f2218a-88f7-4f26-98b3-2d6403bff02d-kube-api-access-9pp9v\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:18.302922 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.302870 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-knv7j" Nov 03 19:39:18.311793 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.311650 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-x9cw5" Nov 03 19:39:18.318656 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.318629 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-d5qxk" Nov 03 19:39:18.337972 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.337941 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:18.346875 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.346849 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:18.350907 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.350885 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-txtq2" Nov 03 19:39:18.358685 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.358654 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4734a1de-ee4e-42ab-b925-1a85f65b7d24-cert\") pod \"ingress-canary-ggbpk\" (UID: \"4734a1de-ee4e-42ab-b925-1a85f65b7d24\") " pod="openshift-ingress-canary/ingress-canary-ggbpk" Nov 03 19:39:18.358804 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.358695 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdcb\" (UniqueName: \"kubernetes.io/projected/4734a1de-ee4e-42ab-b925-1a85f65b7d24-kube-api-access-9hdcb\") pod \"ingress-canary-ggbpk\" (UID: \"4734a1de-ee4e-42ab-b925-1a85f65b7d24\") " pod="openshift-ingress-canary/ingress-canary-ggbpk" Nov 03 19:39:18.360153 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.359993 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:18.363359 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.363333 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4734a1de-ee4e-42ab-b925-1a85f65b7d24-cert\") pod \"ingress-canary-ggbpk\" (UID: \"4734a1de-ee4e-42ab-b925-1a85f65b7d24\") " pod="openshift-ingress-canary/ingress-canary-ggbpk" Nov 03 19:39:18.364921 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.364873 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:18.395683 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.395071 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f9vq4" Nov 03 19:39:18.395683 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.395341 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:39:18.395683 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.395640 2568 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:39:18.399003 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.396912 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:39:18.399647 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.399588 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hdcb\" (UniqueName: \"kubernetes.io/projected/4734a1de-ee4e-42ab-b925-1a85f65b7d24-kube-api-access-9hdcb\") pod \"ingress-canary-ggbpk\" (UID: \"4734a1de-ee4e-42ab-b925-1a85f65b7d24\") " pod="openshift-ingress-canary/ingress-canary-ggbpk" Nov 03 19:39:18.430007 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.429670 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ggbpk" Nov 03 19:39:18.601384 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.601357 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-x9cw5"] Nov 03 19:39:18.603700 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:18.603675 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257528a3_a0f4_4a07_bba0_67561f8ee205.slice/crio-80c68326a1ace6d59c0d6ba6d6ed8a0ef57a6bb8d2d46952cae8742a62c92f2a WatchSource:0}: Error finding container 80c68326a1ace6d59c0d6ba6d6ed8a0ef57a6bb8d2d46952cae8742a62c92f2a: Status 404 returned error can't find the container with id 80c68326a1ace6d59c0d6ba6d6ed8a0ef57a6bb8d2d46952cae8742a62c92f2a Nov 03 19:39:18.606658 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.606640 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-knv7j"] Nov 03 19:39:18.609127 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:18.609100 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e3f7cd_e4ba_44c3_9ed5_2508bb92c041.slice/crio-65a0bf97dec0a5ef6c8f54ba42f048c27928b0d6c5c8e54bcc95c485b11c5a0d WatchSource:0}: Error finding container 65a0bf97dec0a5ef6c8f54ba42f048c27928b0d6c5c8e54bcc95c485b11c5a0d: Status 404 returned error can't find the container with id 65a0bf97dec0a5ef6c8f54ba42f048c27928b0d6c5c8e54bcc95c485b11c5a0d Nov 03 19:39:18.609680 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.609659 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-d5qxk"] Nov 03 19:39:18.616418 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:18.615227 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c48ea1_6b42_4ec0_ae95_bf81b913e901.slice/crio-c717613fd5693f0bdf38ba2177a00d7871e7cc3d3f33a2425911bfb4a5f12f51 WatchSource:0}: Error finding container c717613fd5693f0bdf38ba2177a00d7871e7cc3d3f33a2425911bfb4a5f12f51: Status 404 returned error can't find the container with id c717613fd5693f0bdf38ba2177a00d7871e7cc3d3f33a2425911bfb4a5f12f51 Nov 03 19:39:18.671894 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.671738 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8"] Nov 03 19:39:18.674209 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:18.674171 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c72eb2_3839_4483_82e8_b7d0bf82c57e.slice/crio-81c832b35f800f18e4bd50b3df9a3225e6b10554ef285a4c8222a7a117461f4e WatchSource:0}: Error finding container 81c832b35f800f18e4bd50b3df9a3225e6b10554ef285a4c8222a7a117461f4e: Status 404 returned error can't find the container with id 81c832b35f800f18e4bd50b3df9a3225e6b10554ef285a4c8222a7a117461f4e Nov 03 19:39:18.688040 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.688015 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-rs8wk"] Nov 03 19:39:18.695068 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.695046 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-txtq2"] Nov 03 19:39:18.698331 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:18.698303 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbfc914_8c97_4770_9ef6_6e553c4dcc58.slice/crio-301dc967e308174461e05be646b0c914b2b4be4d8cf4fb6146577310768f20fb WatchSource:0}: Error finding container 301dc967e308174461e05be646b0c914b2b4be4d8cf4fb6146577310768f20fb: Status 404 returned error can't find the container with id 301dc967e308174461e05be646b0c914b2b4be4d8cf4fb6146577310768f20fb Nov 03 19:39:18.699719 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:18.699692 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2266dada_9604_44f9_b488_511006cee7b2.slice/crio-537f6657fe36c1243633fe52bb29c5601741504015ba8f1347d85850aaa3bda2 WatchSource:0}: Error finding container 537f6657fe36c1243633fe52bb29c5601741504015ba8f1347d85850aaa3bda2: Status 404 returned error can't find the container with id 537f6657fe36c1243633fe52bb29c5601741504015ba8f1347d85850aaa3bda2 Nov 03 19:39:18.733239 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.732762 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-f9vq4"] Nov 03 19:39:18.741093 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.739475 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p"] Nov 03 19:39:18.743079 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:18.742887 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d5c2e9_13cd_4859_abbf_218d18877c5a.slice/crio-efa6b54ad3a18df0a75d224a3509f5d0dbd59f5dcbd8423f26f49982c8db8267 WatchSource:0}: Error finding container efa6b54ad3a18df0a75d224a3509f5d0dbd59f5dcbd8423f26f49982c8db8267: Status 404 returned error can't find the container with id efa6b54ad3a18df0a75d224a3509f5d0dbd59f5dcbd8423f26f49982c8db8267 Nov 03 19:39:18.743624 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.743604 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-68j2z"] Nov 03 19:39:18.743891 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:18.743869 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a9a94d_e73f_4942_bab6_efbfcb728df2.slice/crio-e5a1225dc4af688d6dc6d0122badf6e3d0ae85b37475721f59943833a97eff20 WatchSource:0}: Error finding container e5a1225dc4af688d6dc6d0122badf6e3d0ae85b37475721f59943833a97eff20: Status 404 returned error can't find the container with id e5a1225dc4af688d6dc6d0122badf6e3d0ae85b37475721f59943833a97eff20 Nov 03 19:39:18.744221 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:18.744201 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f2218a_88f7_4f26_98b3_2d6403bff02d.slice/crio-6aeb4f12d026ba9495f95d11bfe0501426a34494e2186aa7ef6431675e8818a8 WatchSource:0}: Error finding container 6aeb4f12d026ba9495f95d11bfe0501426a34494e2186aa7ef6431675e8818a8: Status 404 returned error can't find the container with id 6aeb4f12d026ba9495f95d11bfe0501426a34494e2186aa7ef6431675e8818a8 Nov 03 19:39:18.756866 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.756847 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ggbpk"] Nov 03 19:39:18.760446 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:18.760431 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4734a1de_ee4e_42ab_b925_1a85f65b7d24.slice/crio-ee7021019df65ebe84ae81b9ef39ba9cfa640e3b78dce3c8024caaa69c2f2142 WatchSource:0}: Error finding container ee7021019df65ebe84ae81b9ef39ba9cfa640e3b78dce3c8024caaa69c2f2142: Status 404 returned error can't find the container with id ee7021019df65ebe84ae81b9ef39ba9cfa640e3b78dce3c8024caaa69c2f2142 Nov 03 19:39:18.765106 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.765069 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:18.765276 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765258 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle podName:9924fd9e-f7b6-4036-8584-22caa1de39b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:19.76523845 +0000 UTC m=+30.161862499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle") pod "router-default-855cd4cff-prjqj" (UID: "9924fd9e-f7b6-4036-8584-22caa1de39b3") : configmap references non-existent config key: service-ca.crt Nov 03 19:39:18.765363 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.765314 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sk4zs\" (UID: \"b2503201-686a-4cf9-b5a1-977c87e6b717\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:18.765363 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.765352 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:18.765450 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.765409 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:18.765450 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765423 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:39:18.765450 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.765444 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:18.765703 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765462 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert podName:b2503201-686a-4cf9-b5a1-977c87e6b717 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:19.765449929 +0000 UTC m=+30.162073978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sk4zs" (UID: "b2503201-686a-4cf9-b5a1-977c87e6b717") : secret "networking-console-plugin-cert" not found Nov 03 19:39:18.765703 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.765486 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:18.765703 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765512 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:39:18.765703 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.765568 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:18.765703 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765587 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls podName:1b0776e1-a654-4e61-ba94-6d81ddfc8877 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:19.765572846 +0000 UTC m=+30.162196890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-szqck" (UID: "1b0776e1-a654-4e61-ba94-6d81ddfc8877") : secret "samples-operator-tls" not found Nov 03 19:39:18.765703 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:18.765615 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:18.765703 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765621 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:39:18.765703 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765670 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:39:18.766062 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765709 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs podName:9924fd9e-f7b6-4036-8584-22caa1de39b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:19.765697127 +0000 UTC m=+30.162321182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs") pod "router-default-855cd4cff-prjqj" (UID: "9924fd9e-f7b6-4036-8584-22caa1de39b3") : secret "router-metrics-certs-default" not found Nov 03 19:39:18.766062 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765674 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b48f6466f-rvj5f: secret "image-registry-tls" not found Nov 03 19:39:18.766062 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765728 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:39:18.766062 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765740 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls podName:6bab2991-93a7-44c8-9071-e3c971b79885 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:19.765734334 +0000 UTC m=+30.162358405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls") pod "image-registry-b48f6466f-rvj5f" (UID: "6bab2991-93a7-44c8-9071-e3c971b79885") : secret "image-registry-tls" not found Nov 03 19:39:18.766062 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765757 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls podName:ca34d416-e238-48cb-a507-953f78fead22 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:19.765748369 +0000 UTC m=+30.162372397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zx2m4" (UID: "ca34d416-e238-48cb-a507-953f78fead22") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:39:18.766062 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765779 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:39:18.766062 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765787 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85444c64c6-qk7p8: secret "image-registry-tls" not found Nov 03 19:39:18.766062 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765796 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:39:18.766062 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765804 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls podName:ac78f686-0d5a-4ff1-8a04-101e5794b7d7 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:19.765798494 +0000 UTC m=+30.162422560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls") pod "image-registry-85444c64c6-qk7p8" (UID: "ac78f686-0d5a-4ff1-8a04-101e5794b7d7") : secret "image-registry-tls" not found Nov 03 19:39:18.766062 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:18.765815 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls podName:5101a457-b490-4538-a9f6-288014fa9264 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:19.765809246 +0000 UTC m=+30.162433274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls") pod "dns-default-zdf6j" (UID: "5101a457-b490-4538-a9f6-288014fa9264") : secret "dns-default-metrics-tls" not found Nov 03 19:39:19.503417 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.503333 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f9vq4" event={"ID":"09d5c2e9-13cd-4859-abbf-218d18877c5a","Type":"ContainerStarted","Data":"efa6b54ad3a18df0a75d224a3509f5d0dbd59f5dcbd8423f26f49982c8db8267"} Nov 03 19:39:19.506931 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.506768 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rs8wk" event={"ID":"2266dada-9604-44f9-b488-511006cee7b2","Type":"ContainerStarted","Data":"537f6657fe36c1243633fe52bb29c5601741504015ba8f1347d85850aaa3bda2"} Nov 03 19:39:19.511142 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.509982 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-d5qxk" event={"ID":"51c48ea1-6b42-4ec0-ae95-bf81b913e901","Type":"ContainerStarted","Data":"316f2fa9a0c5d3b9637aee7efceb2c9666e77343a982c6c771084f40e3c066cd"} Nov 03 19:39:19.511142 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.510045 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-d5qxk" event={"ID":"51c48ea1-6b42-4ec0-ae95-bf81b913e901","Type":"ContainerStarted","Data":"c717613fd5693f0bdf38ba2177a00d7871e7cc3d3f33a2425911bfb4a5f12f51"} Nov 03 19:39:19.514781 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.513246 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-x9cw5" event={"ID":"257528a3-a0f4-4a07-bba0-67561f8ee205","Type":"ContainerStarted","Data":"221b5f2aa74da47a2a1ff63f3dae5049030aa64d8898d623888b91b67e858f8c"} Nov 03 19:39:19.514781 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.513326 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-x9cw5" event={"ID":"257528a3-a0f4-4a07-bba0-67561f8ee205","Type":"ContainerStarted","Data":"80c68326a1ace6d59c0d6ba6d6ed8a0ef57a6bb8d2d46952cae8742a62c92f2a"} Nov 03 19:39:19.516238 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.515984 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ggbpk" event={"ID":"4734a1de-ee4e-42ab-b925-1a85f65b7d24","Type":"ContainerStarted","Data":"ee7021019df65ebe84ae81b9ef39ba9cfa640e3b78dce3c8024caaa69c2f2142"} Nov 03 19:39:19.520325 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.519922 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-knv7j" event={"ID":"d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041","Type":"ContainerStarted","Data":"34c7f714dc99e01bcc55569430fdbc6b6da210ecdb54592d432cf523ed920803"} Nov 03 19:39:19.520325 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.519970 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-knv7j" event={"ID":"d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041","Type":"ContainerStarted","Data":"65a0bf97dec0a5ef6c8f54ba42f048c27928b0d6c5c8e54bcc95c485b11c5a0d"} Nov 03 19:39:19.524157 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.524125 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" event={"ID":"73f2218a-88f7-4f26-98b3-2d6403bff02d","Type":"ContainerStarted","Data":"6aeb4f12d026ba9495f95d11bfe0501426a34494e2186aa7ef6431675e8818a8"} Nov 03 19:39:19.529886 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.529816 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" event={"ID":"74a9a94d-e73f-4942-bab6-efbfcb728df2","Type":"ContainerStarted","Data":"e5a1225dc4af688d6dc6d0122badf6e3d0ae85b37475721f59943833a97eff20"} Nov 03 19:39:19.531985 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.531901 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-txtq2" event={"ID":"0fbfc914-8c97-4770-9ef6-6e553c4dcc58","Type":"ContainerStarted","Data":"301dc967e308174461e05be646b0c914b2b4be4d8cf4fb6146577310768f20fb"} Nov 03 19:39:19.534181 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.534157 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" event={"ID":"69c72eb2-3839-4483-82e8-b7d0bf82c57e","Type":"ContainerStarted","Data":"81c832b35f800f18e4bd50b3df9a3225e6b10554ef285a4c8222a7a117461f4e"} Nov 03 19:39:19.546135 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.545048 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-d5qxk" podStartSLOduration=414.545031466 podStartE2EDuration="6m54.545031466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:19.527493617 +0000 UTC m=+29.924117667" watchObservedRunningTime="2026-11-03 19:39:19.545031466 +0000 UTC m=+29.941655519" Nov 03 19:39:19.563139 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.562350 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-knv7j" podStartSLOduration=414.562331538 podStartE2EDuration="6m54.562331538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:19.546409791 +0000 UTC m=+29.943033864" watchObservedRunningTime="2026-11-03 19:39:19.562331538 +0000 UTC m=+29.958955591" Nov 03 19:39:19.778567 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.778412 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:19.779575 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.778875 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:39:19.779575 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.778903 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85444c64c6-qk7p8: secret "image-registry-tls" not found Nov 03 19:39:19.779575 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.778971 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls podName:ac78f686-0d5a-4ff1-8a04-101e5794b7d7 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:21.778948281 +0000 UTC m=+32.175572332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls") pod "image-registry-85444c64c6-qk7p8" (UID: "ac78f686-0d5a-4ff1-8a04-101e5794b7d7") : secret "image-registry-tls" not found Nov 03 19:39:19.779575 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.778985 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:39:19.779575 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.779043 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls podName:1b0776e1-a654-4e61-ba94-6d81ddfc8877 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:21.779026123 +0000 UTC m=+32.175650168 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-szqck" (UID: "1b0776e1-a654-4e61-ba94-6d81ddfc8877") : secret "samples-operator-tls" not found Nov 03 19:39:19.779575 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.778887 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:19.779575 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.779079 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:19.779575 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.779176 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:19.779575 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.779229 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:19.779575 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.779289 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:19.779575 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.779339 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sk4zs\" (UID: \"b2503201-686a-4cf9-b5a1-977c87e6b717\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:19.779575 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.779370 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:19.779575 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.779399 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:39:19.779575 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.779415 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b48f6466f-rvj5f: secret "image-registry-tls" not found Nov 03 19:39:19.779575 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.779480 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls podName:6bab2991-93a7-44c8-9071-e3c971b79885 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:21.77946594 +0000 UTC m=+32.176089973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls") pod "image-registry-b48f6466f-rvj5f" (UID: "6bab2991-93a7-44c8-9071-e3c971b79885") : secret "image-registry-tls" not found Nov 03 19:39:19.779575 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.779568 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:39:19.780377 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.779589 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle podName:9924fd9e-f7b6-4036-8584-22caa1de39b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:21.779558098 +0000 UTC m=+32.176182145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle") pod "router-default-855cd4cff-prjqj" (UID: "9924fd9e-f7b6-4036-8584-22caa1de39b3") : configmap references non-existent config key: service-ca.crt Nov 03 19:39:19.780377 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.779613 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs podName:9924fd9e-f7b6-4036-8584-22caa1de39b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:21.779601823 +0000 UTC m=+32.176225868 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs") pod "router-default-855cd4cff-prjqj" (UID: "9924fd9e-f7b6-4036-8584-22caa1de39b3") : secret "router-metrics-certs-default" not found Nov 03 19:39:19.780377 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.779625 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:39:19.780377 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.779677 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls podName:5101a457-b490-4538-a9f6-288014fa9264 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:21.779661867 +0000 UTC m=+32.176285916 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls") pod "dns-default-zdf6j" (UID: "5101a457-b490-4538-a9f6-288014fa9264") : secret "dns-default-metrics-tls" not found Nov 03 19:39:19.780377 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.779715 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:39:19.780377 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.779733 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:39:19.780377 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.779753 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert podName:b2503201-686a-4cf9-b5a1-977c87e6b717 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:21.779740693 +0000 UTC m=+32.176364773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sk4zs" (UID: "b2503201-686a-4cf9-b5a1-977c87e6b717") : secret "networking-console-plugin-cert" not found Nov 03 19:39:19.780377 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:19.779774 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls podName:ca34d416-e238-48cb-a507-953f78fead22 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:21.779762606 +0000 UTC m=+32.176386658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zx2m4" (UID: "ca34d416-e238-48cb-a507-953f78fead22") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:39:19.833245 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.833213 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:19.834586 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.833439 2568 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:39:19.855794 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.855456 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:19.886199 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:19.886119 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-x9cw5" podStartSLOduration=414.88610035 podStartE2EDuration="6m54.88610035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:19.563109968 +0000 UTC m=+29.959734029" watchObservedRunningTime="2026-11-03 19:39:19.88610035 +0000 UTC m=+30.282724401" Nov 03 19:39:21.546009 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:21.545211 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-txtq2" event={"ID":"0fbfc914-8c97-4770-9ef6-6e553c4dcc58","Type":"ContainerStarted","Data":"8ddf4a672ed529c200bae97628f439bef8b8158581f091a953ed57873dd9a88e"} Nov 03 19:39:21.802592 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:21.802498 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:21.802592 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:21.802577 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:21.802832 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:21.802601 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:21.802832 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:21.802677 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:21.802832 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:21.802705 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:21.802832 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:21.802732 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:21.802832 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:21.802773 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sk4zs\" (UID: \"b2503201-686a-4cf9-b5a1-977c87e6b717\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:21.802832 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:21.802794 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:21.803060 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.802934 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:39:21.803060 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.802987 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls podName:ca34d416-e238-48cb-a507-953f78fead22 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:25.802973766 +0000 UTC m=+36.199597807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zx2m4" (UID: "ca34d416-e238-48cb-a507-953f78fead22") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:39:21.803060 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.803041 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:39:21.803060 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.803050 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85444c64c6-qk7p8: secret "image-registry-tls" not found Nov 03 19:39:21.803173 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.803072 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls podName:ac78f686-0d5a-4ff1-8a04-101e5794b7d7 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:25.803065645 +0000 UTC m=+36.199689686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls") pod "image-registry-85444c64c6-qk7p8" (UID: "ac78f686-0d5a-4ff1-8a04-101e5794b7d7") : secret "image-registry-tls" not found Nov 03 19:39:21.803173 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.803123 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:39:21.803173 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.803153 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls podName:1b0776e1-a654-4e61-ba94-6d81ddfc8877 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:25.80314374 +0000 UTC m=+36.199767783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-szqck" (UID: "1b0776e1-a654-4e61-ba94-6d81ddfc8877") : secret "samples-operator-tls" not found Nov 03 19:39:21.803269 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.803190 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:39:21.803269 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.803214 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs podName:9924fd9e-f7b6-4036-8584-22caa1de39b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:25.803204785 +0000 UTC m=+36.199828826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs") pod "router-default-855cd4cff-prjqj" (UID: "9924fd9e-f7b6-4036-8584-22caa1de39b3") : secret "router-metrics-certs-default" not found Nov 03 19:39:21.803269 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.803266 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:39:21.803390 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.803279 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b48f6466f-rvj5f: secret "image-registry-tls" not found Nov 03 19:39:21.803390 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.803311 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls podName:6bab2991-93a7-44c8-9071-e3c971b79885 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:25.80330158 +0000 UTC m=+36.199925622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls") pod "image-registry-b48f6466f-rvj5f" (UID: "6bab2991-93a7-44c8-9071-e3c971b79885") : secret "image-registry-tls" not found Nov 03 19:39:21.803390 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.803376 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:39:21.803491 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.803397 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls podName:5101a457-b490-4538-a9f6-288014fa9264 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:25.803391155 +0000 UTC m=+36.200015183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls") pod "dns-default-zdf6j" (UID: "5101a457-b490-4538-a9f6-288014fa9264") : secret "dns-default-metrics-tls" not found Nov 03 19:39:21.803491 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.803449 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle podName:9924fd9e-f7b6-4036-8584-22caa1de39b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:25.803443118 +0000 UTC m=+36.200067159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle") pod "router-default-855cd4cff-prjqj" (UID: "9924fd9e-f7b6-4036-8584-22caa1de39b3") : configmap references non-existent config key: service-ca.crt Nov 03 19:39:21.803491 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.803488 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:39:21.803650 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:21.803515 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert podName:b2503201-686a-4cf9-b5a1-977c87e6b717 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:25.80350723 +0000 UTC m=+36.200131260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sk4zs" (UID: "b2503201-686a-4cf9-b5a1-977c87e6b717") : secret "networking-console-plugin-cert" not found Nov 03 19:39:22.817057 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:22.817017 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs\") pod \"network-metrics-daemon-z9svh\" (UID: \"2dcde652-ceea-4a58-9746-eb2087dea702\") " pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:22.817742 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:22.817179 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:22.817742 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:22.817189 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 19:39:22.817742 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:22.817267 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs podName:2dcde652-ceea-4a58-9746-eb2087dea702 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:54.817244996 +0000 UTC m=+65.213869025 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs") pod "network-metrics-daemon-z9svh" (UID: "2dcde652-ceea-4a58-9746-eb2087dea702") : secret "metrics-daemon-secret" not found Nov 03 19:39:22.821892 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:22.821861 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:23.019323 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.019150 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22vq\" (UniqueName: \"kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq\") pod \"network-check-target-wztqw\" (UID: \"d85c1bad-d10e-4276-9471-0fbe78bb67b3\") " pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:23.021978 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.021955 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k22vq\" (UniqueName: \"kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq\") pod \"network-check-target-wztqw\" (UID: \"d85c1bad-d10e-4276-9471-0fbe78bb67b3\") " pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:23.032012 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.031986 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:23.318769 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.318731 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:23.552914 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.552878 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" event={"ID":"69c72eb2-3839-4483-82e8-b7d0bf82c57e","Type":"ContainerStarted","Data":"d2ce890fb0dbd1dc435e63386ece22b4c82ba8b7f034c3c18d6aed9c9e152512"} Nov 03 19:39:23.569482 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.569412 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" podStartSLOduration=383.376490975 podStartE2EDuration="6m27.569398944s" totalImagesPullingTime="4.192907969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:23.569097047 +0000 UTC m=+33.965721100" watchObservedRunningTime="2026-11-03 19:39:23.569398944 +0000 UTC m=+33.966022994" Nov 03 19:39:23.569672 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.569647 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-txtq2" podStartSLOduration=384.021516077 podStartE2EDuration="6m26.569640318s" totalImagesPullingTime="2.548124241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:21.562401327 +0000 UTC m=+31.959025377" watchObservedRunningTime="2026-11-03 19:39:23.569640318 +0000 UTC m=+33.966264370" Nov 03 19:39:23.714831 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.714004 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p4t7z"] Nov 03 19:39:23.716132 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.716033 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wztqw"] Nov 03 19:39:23.718764 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:23.718739 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedae5b46_8158_4dc8_91c9_06ed65126db9.slice/crio-6fcfaa3b721fe1f6c6f44e7656bfdd504a700ef4461095947f0ca7a98333d41b WatchSource:0}: Error finding container 6fcfaa3b721fe1f6c6f44e7656bfdd504a700ef4461095947f0ca7a98333d41b: Status 404 returned error can't find the container with id 6fcfaa3b721fe1f6c6f44e7656bfdd504a700ef4461095947f0ca7a98333d41b Nov 03 19:39:23.817931 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:23.817903 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85c1bad_d10e_4276_9471_0fbe78bb67b3.slice/crio-0657357d1926ac615b5c46eed0196a7b0015b1c6842d7c136771bd5a6fddebfe WatchSource:0}: Error finding container 0657357d1926ac615b5c46eed0196a7b0015b1c6842d7c136771bd5a6fddebfe: Status 404 returned error can't find the container with id 0657357d1926ac615b5c46eed0196a7b0015b1c6842d7c136771bd5a6fddebfe Nov 03 19:39:23.846324 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.833109 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-fckg5"] Nov 03 19:39:23.849118 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.849096 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:23.850218 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.850159 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-fckg5"] Nov 03 19:39:23.853445 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.853425 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 19:39:23.853583 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.853511 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 19:39:23.853583 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.853527 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 19:39:23.853749 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.853636 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 19:39:23.853749 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.853698 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jhn8f\"" Nov 03 19:39:23.935971 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.935943 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwnxx\" (UniqueName: \"kubernetes.io/projected/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-kube-api-access-hwnxx\") pod \"service-ca-5fffc97bdf-fckg5\" (UID: \"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:23.936134 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.936060 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-signing-cabundle\") pod \"service-ca-5fffc97bdf-fckg5\" (UID: \"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:23.936211 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:23.936189 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-signing-key\") pod \"service-ca-5fffc97bdf-fckg5\" (UID: \"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:24.037425 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.037364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-signing-cabundle\") pod \"service-ca-5fffc97bdf-fckg5\" (UID: \"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:24.037624 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.037489 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-signing-key\") pod \"service-ca-5fffc97bdf-fckg5\" (UID: \"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:24.037669 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.037633 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwnxx\" (UniqueName: \"kubernetes.io/projected/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-kube-api-access-hwnxx\") pod \"service-ca-5fffc97bdf-fckg5\" (UID: \"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:24.038403 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.038370 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-signing-cabundle\") pod \"service-ca-5fffc97bdf-fckg5\" (UID: \"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:24.040179 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.040155 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-signing-key\") pod \"service-ca-5fffc97bdf-fckg5\" (UID: \"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:24.045728 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.045694 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwnxx\" (UniqueName: \"kubernetes.io/projected/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-kube-api-access-hwnxx\") pod \"service-ca-5fffc97bdf-fckg5\" (UID: \"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:24.168872 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.168841 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-slmpf"] Nov 03 19:39:24.181334 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.181308 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-slmpf" Nov 03 19:39:24.186371 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.186347 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:24.191880 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.191342 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-slmpf"] Nov 03 19:39:24.191880 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.191703 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 19:39:24.191880 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.191764 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 19:39:24.191880 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.191853 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-56c2j\"" Nov 03 19:39:24.199826 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.199807 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-t8rfb"] Nov 03 19:39:24.215063 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.215028 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t8rfb" Nov 03 19:39:24.219993 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.219969 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-t8rfb"] Nov 03 19:39:24.239480 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.239451 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4bgf\" (UniqueName: \"kubernetes.io/projected/0813f12f-0d14-47a8-b4b0-76b159b4e845-kube-api-access-v4bgf\") pod \"migrator-66df5d496b-t8rfb\" (UID: \"0813f12f-0d14-47a8-b4b0-76b159b4e845\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t8rfb" Nov 03 19:39:24.239664 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.239513 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jz64\" (UniqueName: \"kubernetes.io/projected/6efc8dc6-6157-4518-b5fc-dfa8af652691-kube-api-access-2jz64\") pod \"migrator-66df5d496b-slmpf\" (UID: \"6efc8dc6-6157-4518-b5fc-dfa8af652691\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-slmpf" Nov 03 19:39:24.272164 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.272143 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-md5d2"] Nov 03 19:39:24.290585 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.290562 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-fn7j5-kgmpq"] Nov 03 19:39:24.290714 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.290636 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:24.294426 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.294401 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 19:39:24.294728 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.294710 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 19:39:24.294989 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.294749 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7hnqg\"" Nov 03 19:39:24.311255 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.311115 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-md5d2"] Nov 03 19:39:24.311255 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.311154 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fn7j5-kgmpq"] Nov 03 19:39:24.311645 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.311391 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:24.340878 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.340836 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-archives-path\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:24.341022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.340881 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zcvd\" (UniqueName: \"kubernetes.io/projected/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-kube-api-access-7zcvd\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:24.341022 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.340993 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:24.341104 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.341025 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4bgf\" (UniqueName: \"kubernetes.io/projected/0813f12f-0d14-47a8-b4b0-76b159b4e845-kube-api-access-v4bgf\") pod \"migrator-66df5d496b-t8rfb\" (UID: \"0813f12f-0d14-47a8-b4b0-76b159b4e845\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t8rfb" Nov 03 19:39:24.341104 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.341056 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-trusted-ca-bundle\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:24.341235 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.341223 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d134b335-9e0e-41c9-a676-566a6ef55f5a-crio-socket\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:24.341275 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.341267 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jz64\" (UniqueName: \"kubernetes.io/projected/6efc8dc6-6157-4518-b5fc-dfa8af652691-kube-api-access-2jz64\") pod \"migrator-66df5d496b-slmpf\" (UID: \"6efc8dc6-6157-4518-b5fc-dfa8af652691\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-slmpf" Nov 03 19:39:24.341371 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.341353 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hnsk\" (UniqueName: \"kubernetes.io/projected/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-api-access-8hnsk\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:24.341425 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.341381 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-service-ca-bundle\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:24.341425 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.341409 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d134b335-9e0e-41c9-a676-566a6ef55f5a-data-volume\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:24.341562 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.341542 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:24.349308 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.349284 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4bgf\" (UniqueName: \"kubernetes.io/projected/0813f12f-0d14-47a8-b4b0-76b159b4e845-kube-api-access-v4bgf\") pod \"migrator-66df5d496b-t8rfb\" (UID: \"0813f12f-0d14-47a8-b4b0-76b159b4e845\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t8rfb" Nov 03 19:39:24.349308 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.349301 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jz64\" (UniqueName: \"kubernetes.io/projected/6efc8dc6-6157-4518-b5fc-dfa8af652691-kube-api-access-2jz64\") pod \"migrator-66df5d496b-slmpf\" (UID: \"6efc8dc6-6157-4518-b5fc-dfa8af652691\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-slmpf" Nov 03 19:39:24.442218 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.442187 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-trusted-ca-bundle\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:24.442218 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.442219 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d134b335-9e0e-41c9-a676-566a6ef55f5a-crio-socket\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:24.442388 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.442276 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hnsk\" (UniqueName: \"kubernetes.io/projected/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-api-access-8hnsk\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:24.442388 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.442293 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-service-ca-bundle\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:24.442493 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.442382 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d134b335-9e0e-41c9-a676-566a6ef55f5a-crio-socket\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:24.442493 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.442447 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d134b335-9e0e-41c9-a676-566a6ef55f5a-data-volume\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:24.442622 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.442549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:24.442668 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.442618 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-archives-path\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:24.442717 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.442666 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zcvd\" (UniqueName: \"kubernetes.io/projected/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-kube-api-access-7zcvd\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:24.442814 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.442761 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:24.442900 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:24.442881 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:39:24.442955 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.442933 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-service-ca-bundle\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:24.442955 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:24.442941 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls podName:d134b335-9e0e-41c9-a676-566a6ef55f5a nodeName:}" failed. No retries permitted until 2026-11-03 19:39:24.942924441 +0000 UTC m=+35.339548484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-md5d2" (UID: "d134b335-9e0e-41c9-a676-566a6ef55f5a") : secret "insights-runtime-extractor-tls" not found Nov 03 19:39:24.443070 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:24.442940 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:39:24.443070 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:24.443000 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm podName:d134b335-9e0e-41c9-a676-566a6ef55f5a nodeName:}" failed. No retries permitted until 2026-11-03 19:39:24.942989257 +0000 UTC m=+35.339613299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-md5d2" (UID: "d134b335-9e0e-41c9-a676-566a6ef55f5a") : configmap "kube-rbac-proxy" not found Nov 03 19:39:24.443212 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.443190 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d134b335-9e0e-41c9-a676-566a6ef55f5a-data-volume\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:24.443400 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.443380 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-archives-path\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:24.443560 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.443525 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-trusted-ca-bundle\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:24.453346 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.453321 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hnsk\" (UniqueName: \"kubernetes.io/projected/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-api-access-8hnsk\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:24.453432 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.453404 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zcvd\" (UniqueName: \"kubernetes.io/projected/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-kube-api-access-7zcvd\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:24.501664 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.501637 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-slmpf" Nov 03 19:39:24.528215 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.528188 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-fckg5"] Nov 03 19:39:24.530844 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:24.530815 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd367f3_a564_4922_8d7b_2d4b84eaa1bc.slice/crio-58179311e88b6bacd355fcb0851db9a643bc8ac75da54726b904130e3ad6f754 WatchSource:0}: Error finding container 58179311e88b6bacd355fcb0851db9a643bc8ac75da54726b904130e3ad6f754: Status 404 returned error can't find the container with id 58179311e88b6bacd355fcb0851db9a643bc8ac75da54726b904130e3ad6f754 Nov 03 19:39:24.537225 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.537199 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t8rfb" Nov 03 19:39:24.560491 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.560356 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" event={"ID":"74a9a94d-e73f-4942-bab6-efbfcb728df2","Type":"ContainerStarted","Data":"940af9c301b83dbaad9deab8529a9b1e3eaaf3716f05ccf096c0250869b9891e"} Nov 03 19:39:24.578627 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.578587 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p4t7z" event={"ID":"edae5b46-8158-4dc8-91c9-06ed65126db9","Type":"ContainerStarted","Data":"6fcfaa3b721fe1f6c6f44e7656bfdd504a700ef4461095947f0ca7a98333d41b"} Nov 03 19:39:24.580768 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.580618 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" podStartSLOduration=383.919663296 podStartE2EDuration="6m28.580601966s" totalImagesPullingTime="4.66093867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:24.580420239 +0000 UTC m=+34.977044291" watchObservedRunningTime="2026-11-03 19:39:24.580601966 +0000 UTC m=+34.977226017" Nov 03 19:39:24.582432 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.582403 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f9vq4" event={"ID":"09d5c2e9-13cd-4859-abbf-218d18877c5a","Type":"ContainerStarted","Data":"dfca3b31b9b509db8c3666450f49ba75214880d20da0d127ab8dc821d981cc6e"} Nov 03 19:39:24.601104 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.601057 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f9vq4" podStartSLOduration=378.076142078 podStartE2EDuration="6m23.601039725s" totalImagesPullingTime="5.524897647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:24.59986964 +0000 UTC m=+34.996493690" watchObservedRunningTime="2026-11-03 19:39:24.601039725 +0000 UTC m=+34.997663775" Nov 03 19:39:24.602504 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.602484 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rs8wk" event={"ID":"2266dada-9604-44f9-b488-511006cee7b2","Type":"ContainerStarted","Data":"a5f6e3b2906b45fcc933c819d7b6e6f3a27347f3e9b920717d7c5b413bb181cd"} Nov 03 19:39:24.607730 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.607655 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ggbpk" event={"ID":"4734a1de-ee4e-42ab-b925-1a85f65b7d24","Type":"ContainerStarted","Data":"3c8d711080e3ff2ff483dd8cb1bfc9679bf974aced4d6a2b5bcedd98a36e320d"} Nov 03 19:39:24.610026 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.609654 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/0.log" Nov 03 19:39:24.610026 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.609694 2568 generic.go:356] "Generic (PLEG): container finished" podID="73f2218a-88f7-4f26-98b3-2d6403bff02d" containerID="f4c0b389326f66b4b112ab6be2737d2a9e590615fc850dc8953f53aed4d76825" exitCode=255 Nov 03 19:39:24.610026 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.609858 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" event={"ID":"73f2218a-88f7-4f26-98b3-2d6403bff02d","Type":"ContainerDied","Data":"f4c0b389326f66b4b112ab6be2737d2a9e590615fc850dc8953f53aed4d76825"} Nov 03 19:39:24.610026 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.610022 2568 scope.go:122] "RemoveContainer" containerID="f4c0b389326f66b4b112ab6be2737d2a9e590615fc850dc8953f53aed4d76825" Nov 03 19:39:24.615570 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.615355 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" event={"ID":"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc","Type":"ContainerStarted","Data":"58179311e88b6bacd355fcb0851db9a643bc8ac75da54726b904130e3ad6f754"} Nov 03 19:39:24.619060 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.618940 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wztqw" event={"ID":"d85c1bad-d10e-4276-9471-0fbe78bb67b3","Type":"ContainerStarted","Data":"b4c85bfb1fc5f8ac2796bb9c485e4d6a42d3332e5f13894e692e1804801b089c"} Nov 03 19:39:24.619060 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.618981 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wztqw" event={"ID":"d85c1bad-d10e-4276-9471-0fbe78bb67b3","Type":"ContainerStarted","Data":"0657357d1926ac615b5c46eed0196a7b0015b1c6842d7c136771bd5a6fddebfe"} Nov 03 19:39:24.619060 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.619029 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:24.624751 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.624677 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-rs8wk" podStartSLOduration=383.931191474 podStartE2EDuration="6m28.624661916s" totalImagesPullingTime="4.693470442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:24.623130696 +0000 UTC m=+35.019754745" watchObservedRunningTime="2026-11-03 19:39:24.624661916 +0000 UTC m=+35.021285968" Nov 03 19:39:24.656719 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.656691 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:24.661076 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.660907 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wztqw" podStartSLOduration=34.074690362 podStartE2EDuration="34.66088888s" totalImagesPullingTime="586.198518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:24.659290949 +0000 UTC m=+35.055915001" watchObservedRunningTime="2026-11-03 19:39:24.66088888 +0000 UTC m=+35.057512931" Nov 03 19:39:24.680029 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.679974 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ggbpk" podStartSLOduration=2.588370114 podStartE2EDuration="7.679954004s" totalImagesPullingTime="5.09158389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:24.678790999 +0000 UTC m=+35.075415048" watchObservedRunningTime="2026-11-03 19:39:24.679954004 +0000 UTC m=+35.076578055" Nov 03 19:39:24.797214 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.796838 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-slmpf"] Nov 03 19:39:24.799479 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:24.799431 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6efc8dc6_6157_4518_b5fc_dfa8af652691.slice/crio-5e228e40d159d59cf446c8b08af554e1ad50de74c5568e9ce9e3b42ff3ddb0ee WatchSource:0}: Error finding container 5e228e40d159d59cf446c8b08af554e1ad50de74c5568e9ce9e3b42ff3ddb0ee: Status 404 returned error can't find the container with id 5e228e40d159d59cf446c8b08af554e1ad50de74c5568e9ce9e3b42ff3ddb0ee Nov 03 19:39:24.855544 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.855503 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-t8rfb"] Nov 03 19:39:24.860690 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:24.860651 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0813f12f_0d14_47a8_b4b0_76b159b4e845.slice/crio-334a4a410b86b9268850049fee7f11d2c1f8958909cc2287bbba5788176f610c WatchSource:0}: Error finding container 334a4a410b86b9268850049fee7f11d2c1f8958909cc2287bbba5788176f610c: Status 404 returned error can't find the container with id 334a4a410b86b9268850049fee7f11d2c1f8958909cc2287bbba5788176f610c Nov 03 19:39:24.952843 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.952767 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:24.953056 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:24.952870 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:39:24.953056 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.952880 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:24.953056 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:24.952945 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm podName:d134b335-9e0e-41c9-a676-566a6ef55f5a nodeName:}" failed. No retries permitted until 2026-11-03 19:39:25.952923594 +0000 UTC m=+36.349547638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-md5d2" (UID: "d134b335-9e0e-41c9-a676-566a6ef55f5a") : configmap "kube-rbac-proxy" not found Nov 03 19:39:24.956979 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:24.953122 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:39:24.956979 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:24.953192 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls podName:d134b335-9e0e-41c9-a676-566a6ef55f5a nodeName:}" failed. No retries permitted until 2026-11-03 19:39:25.95316528 +0000 UTC m=+36.349789321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-md5d2" (UID: "d134b335-9e0e-41c9-a676-566a6ef55f5a") : secret "insights-runtime-extractor-tls" not found Nov 03 19:39:24.969379 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:24.969355 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fn7j5-kgmpq"] Nov 03 19:39:24.972671 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:24.972640 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e1e0d8_a4ca_41be_a0cf_c3d2da4d7467.slice/crio-5a8d093e45f6dd57c6ae8973495db5795a1a62dded9a37a264bc8eb64d3b0ee7 WatchSource:0}: Error finding container 5a8d093e45f6dd57c6ae8973495db5795a1a62dded9a37a264bc8eb64d3b0ee7: Status 404 returned error can't find the container with id 5a8d093e45f6dd57c6ae8973495db5795a1a62dded9a37a264bc8eb64d3b0ee7 Nov 03 19:39:25.625358 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.625280 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-slmpf" event={"ID":"6efc8dc6-6157-4518-b5fc-dfa8af652691","Type":"ContainerStarted","Data":"5e228e40d159d59cf446c8b08af554e1ad50de74c5568e9ce9e3b42ff3ddb0ee"} Nov 03 19:39:25.628667 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.628593 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t8rfb" event={"ID":"0813f12f-0d14-47a8-b4b0-76b159b4e845","Type":"ContainerStarted","Data":"334a4a410b86b9268850049fee7f11d2c1f8958909cc2287bbba5788176f610c"} Nov 03 19:39:25.632202 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.631668 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" event={"ID":"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467","Type":"ContainerStarted","Data":"891f85911a5b5604b88db6cc45b9acc1fc4aab5697e26a3f2ca57a763135c4f3"} Nov 03 19:39:25.632202 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.631713 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" event={"ID":"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467","Type":"ContainerStarted","Data":"5a8d093e45f6dd57c6ae8973495db5795a1a62dded9a37a264bc8eb64d3b0ee7"} Nov 03 19:39:25.642928 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.642749 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 19:39:25.644868 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.643769 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/0.log" Nov 03 19:39:25.644868 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.643802 2568 generic.go:356] "Generic (PLEG): container finished" podID="73f2218a-88f7-4f26-98b3-2d6403bff02d" containerID="cc6f5944bc7c8c6ba63cc9c0e8ce8c89768dca1e2ed1b8fc6ea3065c53dfeb5e" exitCode=255 Nov 03 19:39:25.644868 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.643865 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" event={"ID":"73f2218a-88f7-4f26-98b3-2d6403bff02d","Type":"ContainerDied","Data":"cc6f5944bc7c8c6ba63cc9c0e8ce8c89768dca1e2ed1b8fc6ea3065c53dfeb5e"} Nov 03 19:39:25.644868 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.643918 2568 scope.go:122] "RemoveContainer" containerID="f4c0b389326f66b4b112ab6be2737d2a9e590615fc850dc8953f53aed4d76825" Nov 03 19:39:25.644868 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.644284 2568 scope.go:122] "RemoveContainer" containerID="cc6f5944bc7c8c6ba63cc9c0e8ce8c89768dca1e2ed1b8fc6ea3065c53dfeb5e" Nov 03 19:39:25.644868 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.644574 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-68j2z_openshift-console-operator(73f2218a-88f7-4f26-98b3-2d6403bff02d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" podUID="73f2218a-88f7-4f26-98b3-2d6403bff02d" Nov 03 19:39:25.650637 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.650411 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" event={"ID":"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc","Type":"ContainerStarted","Data":"8ed7fc235659f385857246310c827e1882bc22de8390b0f619ee53c9ed933c1f"} Nov 03 19:39:25.681160 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.681113 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" podStartSLOduration=1.681093843 podStartE2EDuration="1.681093843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:25.658729213 +0000 UTC m=+36.055353263" watchObservedRunningTime="2026-11-03 19:39:25.681093843 +0000 UTC m=+36.077717894" Nov 03 19:39:25.711621 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.711065 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" podStartSLOduration=2.711045591 podStartE2EDuration="2.711045591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:25.689954303 +0000 UTC m=+36.086578354" watchObservedRunningTime="2026-11-03 19:39:25.711045591 +0000 UTC m=+36.107669647" Nov 03 19:39:25.863826 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.863787 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:25.864301 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.863853 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sk4zs\" (UID: \"b2503201-686a-4cf9-b5a1-977c87e6b717\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:25.864301 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.863883 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:25.864301 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.864036 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:25.864301 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.864051 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:39:25.864301 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.864097 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:25.864301 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.864129 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:25.864301 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.864147 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert podName:b2503201-686a-4cf9-b5a1-977c87e6b717 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:33.864124955 +0000 UTC m=+44.260749009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sk4zs" (UID: "b2503201-686a-4cf9-b5a1-977c87e6b717") : secret "networking-console-plugin-cert" not found Nov 03 19:39:25.864301 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.864195 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:39:25.864301 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.864245 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:25.864301 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.864258 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs podName:9924fd9e-f7b6-4036-8584-22caa1de39b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:33.864242256 +0000 UTC m=+44.260866299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs") pod "router-default-855cd4cff-prjqj" (UID: "9924fd9e-f7b6-4036-8584-22caa1de39b3") : secret "router-metrics-certs-default" not found Nov 03 19:39:25.864301 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.864285 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:25.864889 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.864317 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:39:25.864889 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.864329 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b48f6466f-rvj5f: secret "image-registry-tls" not found Nov 03 19:39:25.864889 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.864367 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls podName:6bab2991-93a7-44c8-9071-e3c971b79885 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:33.864353881 +0000 UTC m=+44.260977932 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls") pod "image-registry-b48f6466f-rvj5f" (UID: "6bab2991-93a7-44c8-9071-e3c971b79885") : secret "image-registry-tls" not found Nov 03 19:39:25.864889 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.864374 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:39:25.864889 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.864406 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls podName:5101a457-b490-4538-a9f6-288014fa9264 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:33.864394826 +0000 UTC m=+44.261018870 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls") pod "dns-default-zdf6j" (UID: "5101a457-b490-4538-a9f6-288014fa9264") : secret "dns-default-metrics-tls" not found Nov 03 19:39:25.864889 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.864453 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle podName:9924fd9e-f7b6-4036-8584-22caa1de39b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:33.864443465 +0000 UTC m=+44.261067520 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle") pod "router-default-855cd4cff-prjqj" (UID: "9924fd9e-f7b6-4036-8584-22caa1de39b3") : configmap references non-existent config key: service-ca.crt Nov 03 19:39:25.864889 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.864511 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:39:25.864889 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.864525 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85444c64c6-qk7p8: secret "image-registry-tls" not found Nov 03 19:39:25.864889 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.864608 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls podName:ac78f686-0d5a-4ff1-8a04-101e5794b7d7 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:33.864577464 +0000 UTC m=+44.261201510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls") pod "image-registry-85444c64c6-qk7p8" (UID: "ac78f686-0d5a-4ff1-8a04-101e5794b7d7") : secret "image-registry-tls" not found Nov 03 19:39:25.864889 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.864606 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:39:25.864889 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.864710 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls podName:1b0776e1-a654-4e61-ba94-6d81ddfc8877 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:33.86469947 +0000 UTC m=+44.261323521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-szqck" (UID: "1b0776e1-a654-4e61-ba94-6d81ddfc8877") : secret "samples-operator-tls" not found Nov 03 19:39:25.874613 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.874583 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:25.929552 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.929452 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:25.965257 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.965221 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:25.965398 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:25.965321 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:25.965605 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.965575 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:39:25.965729 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.965642 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls podName:d134b335-9e0e-41c9-a676-566a6ef55f5a nodeName:}" failed. No retries permitted until 2026-11-03 19:39:27.965624436 +0000 UTC m=+38.362248482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-md5d2" (UID: "d134b335-9e0e-41c9-a676-566a6ef55f5a") : secret "insights-runtime-extractor-tls" not found Nov 03 19:39:25.965985 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.965964 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:39:25.966069 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:25.966018 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm podName:d134b335-9e0e-41c9-a676-566a6ef55f5a nodeName:}" failed. No retries permitted until 2026-11-03 19:39:27.966004542 +0000 UTC m=+38.362628588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-md5d2" (UID: "d134b335-9e0e-41c9-a676-566a6ef55f5a") : configmap "kube-rbac-proxy" not found Nov 03 19:39:26.249641 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:26.249563 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4"] Nov 03 19:39:26.254791 ip-10-0-134-178 kubenswrapper[2568]: W1103 19:39:26.254740 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca34d416_e238_48cb_a507_953f78fead22.slice/crio-defefae7588236ad1b108973ce917842a3db6cd60a9b836c99c6637dad7e8b09 WatchSource:0}: Error finding container defefae7588236ad1b108973ce917842a3db6cd60a9b836c99c6637dad7e8b09: Status 404 returned error can't find the container with id defefae7588236ad1b108973ce917842a3db6cd60a9b836c99c6637dad7e8b09 Nov 03 19:39:26.657248 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:26.657209 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" event={"ID":"ca34d416-e238-48cb-a507-953f78fead22","Type":"ContainerStarted","Data":"defefae7588236ad1b108973ce917842a3db6cd60a9b836c99c6637dad7e8b09"} Nov 03 19:39:26.658986 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:26.658966 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 19:39:26.659569 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:26.659525 2568 scope.go:122] "RemoveContainer" containerID="cc6f5944bc7c8c6ba63cc9c0e8ce8c89768dca1e2ed1b8fc6ea3065c53dfeb5e" Nov 03 19:39:26.659742 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:26.659716 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-68j2z_openshift-console-operator(73f2218a-88f7-4f26-98b3-2d6403bff02d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" podUID="73f2218a-88f7-4f26-98b3-2d6403bff02d" Nov 03 19:39:27.665401 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:27.665361 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-slmpf" event={"ID":"6efc8dc6-6157-4518-b5fc-dfa8af652691","Type":"ContainerStarted","Data":"070cc5ad058bf825e85ccf52b9c77e1e229ea6bee046b71c43086bd91ec6e492"} Nov 03 19:39:27.668131 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:27.668102 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t8rfb" event={"ID":"0813f12f-0d14-47a8-b4b0-76b159b4e845","Type":"ContainerStarted","Data":"39a0103e6fa7f0c6afaca45a10701fbb3023179cf48d32bc0147f7f3bed63e3b"} Nov 03 19:39:27.681738 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:27.681696 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-slmpf" podStartSLOduration=1.569624457 podStartE2EDuration="3.681684721s" totalImagesPullingTime="2.112060264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:27.681176875 +0000 UTC m=+38.077800925" watchObservedRunningTime="2026-11-03 19:39:27.681684721 +0000 UTC m=+38.078308771" Nov 03 19:39:27.987117 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:27.987039 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:27.987288 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:27.987196 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:27.987353 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:27.987325 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:39:27.987413 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:27.987393 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm podName:d134b335-9e0e-41c9-a676-566a6ef55f5a nodeName:}" failed. No retries permitted until 2026-11-03 19:39:31.987372352 +0000 UTC m=+42.383996381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-md5d2" (UID: "d134b335-9e0e-41c9-a676-566a6ef55f5a") : configmap "kube-rbac-proxy" not found Nov 03 19:39:27.987672 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:27.987564 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:39:27.987672 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:27.987642 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls podName:d134b335-9e0e-41c9-a676-566a6ef55f5a nodeName:}" failed. No retries permitted until 2026-11-03 19:39:31.987621561 +0000 UTC m=+42.384245612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-md5d2" (UID: "d134b335-9e0e-41c9-a676-566a6ef55f5a") : secret "insights-runtime-extractor-tls" not found Nov 03 19:39:28.366150 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:28.365309 2568 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:28.366150 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:28.365413 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:28.366150 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:28.365824 2568 scope.go:122] "RemoveContainer" containerID="cc6f5944bc7c8c6ba63cc9c0e8ce8c89768dca1e2ed1b8fc6ea3065c53dfeb5e" Nov 03 19:39:28.366150 ip-10-0-134-178 kubenswrapper[2568]: E1103 19:39:28.366054 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-68j2z_openshift-console-operator(73f2218a-88f7-4f26-98b3-2d6403bff02d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" podUID="73f2218a-88f7-4f26-98b3-2d6403bff02d" Nov 03 19:39:29.678823 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:29.678787 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" event={"ID":"ca34d416-e238-48cb-a507-953f78fead22","Type":"ContainerStarted","Data":"ec192b432cd90e8bff7864080cb2d052146a8b594dca731ee4a36e4608696a85"} Nov 03 19:39:29.697162 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:29.697099 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" podStartSLOduration=387.056140068 podStartE2EDuration="6m29.697086081s" totalImagesPullingTime="2.640946013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:29.695167127 +0000 UTC m=+40.091791177" watchObservedRunningTime="2026-11-03 19:39:29.697086081 +0000 UTC m=+40.093710130" Nov 03 19:39:29.697734 ip-10-0-134-178 kubenswrapper[2568]: I1103 19:39:29.697703 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t8rfb" podStartSLOduration=3.647539625 podStartE2EDuration="5.697691392s" totalImagesPullingTime="2.050151767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:27.697560937 +0000 UTC m=+38.094184990" watchObservedRunningTime="2026-11-03 19:39:29.697691392 +0000 UTC m=+40.094315487" Nov 03 19:39:29.977257 ip-10-0-134-178 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 19:39:29.995013 ip-10-0-134-178 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 19:39:29.995198 ip-10-0-134-178 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 19:39:29.995371 ip-10-0-134-178 systemd[1]: kubelet.service: Consumed 3.107s CPU time. Nov 03 19:39:30.004327 ip-10-0-134-178 systemd[1]: Starting Kubernetes Kubelet... Nov 03 19:39:30.082038 ip-10-0-134-178 kubenswrapper[5111]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:39:30.082038 ip-10-0-134-178 kubenswrapper[5111]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 19:39:30.082038 ip-10-0-134-178 kubenswrapper[5111]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:39:30.082038 ip-10-0-134-178 kubenswrapper[5111]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:39:30.084635 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084527 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:39:30.084635 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084631 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:39:30.084635 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084636 5111 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:39:30.084635 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084641 5111 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:39:30.084635 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084645 5111 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084649 5111 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084653 5111 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084663 5111 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084667 5111 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084673 5111 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084677 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084681 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084685 5111 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084688 5111 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084692 5111 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084696 5111 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084700 5111 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084704 5111 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084707 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084711 5111 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084715 5111 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084719 5111 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084723 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084728 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:39:30.084991 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084732 5111 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084736 5111 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084740 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084744 5111 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084748 5111 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084752 5111 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084756 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084760 5111 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084764 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084768 5111 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084772 5111 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084776 5111 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084780 5111 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084784 5111 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084790 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084795 5111 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084800 5111 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084804 5111 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084808 5111 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:39:30.085889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084813 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084817 5111 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084821 5111 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084826 5111 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084830 5111 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084835 5111 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084840 5111 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084844 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084849 5111 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084853 5111 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084857 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084862 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084867 5111 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084872 5111 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084876 5111 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084880 5111 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084915 5111 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084922 5111 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084928 5111 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084932 5111 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:39:30.086773 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084937 5111 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084941 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084946 5111 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084951 5111 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084955 5111 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084961 5111 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084966 5111 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084970 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084975 5111 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084980 5111 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084984 5111 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084991 5111 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.084995 5111 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085010 5111 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085015 5111 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085019 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085024 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085028 5111 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085033 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:39:30.087716 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085037 5111 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085042 5111 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085046 5111 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085051 5111 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085055 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085061 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085066 5111 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085071 5111 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085075 5111 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085082 5111 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085089 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085094 5111 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085100 5111 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085104 5111 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085109 5111 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085113 5111 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085119 5111 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085123 5111 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085128 5111 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:39:30.088609 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085132 5111 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085137 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085141 5111 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085146 5111 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.085154 5111 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086465 5111 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086476 5111 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086481 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086485 5111 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086489 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086494 5111 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086499 5111 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086503 5111 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086508 5111 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086512 5111 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086516 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086521 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086547 5111 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086552 5111 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:39:30.089480 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086557 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086561 5111 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086566 5111 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086570 5111 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086574 5111 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086579 5111 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086583 5111 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086587 5111 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086591 5111 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086596 5111 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086600 5111 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086606 5111 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086610 5111 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086615 5111 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086620 5111 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086624 5111 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086628 5111 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086638 5111 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086642 5111 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086647 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:39:30.090407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086651 5111 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086655 5111 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086660 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086667 5111 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086673 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086679 5111 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086685 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086690 5111 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086695 5111 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086700 5111 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086704 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086709 5111 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086714 5111 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086718 5111 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086723 5111 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086727 5111 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086732 5111 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086737 5111 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086741 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:39:30.091282 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086746 5111 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086750 5111 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086755 5111 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086759 5111 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086763 5111 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086768 5111 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086772 5111 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086777 5111 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086781 5111 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086785 5111 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086791 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086795 5111 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086802 5111 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086809 5111 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086814 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086819 5111 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086824 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086828 5111 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086833 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:39:30.091830 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086838 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086842 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086847 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086852 5111 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086856 5111 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086861 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086865 5111 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086869 5111 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086874 5111 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086878 5111 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086882 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086887 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086891 5111 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086895 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086899 5111 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086904 5111 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086908 5111 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086916 5111 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086920 5111 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086925 5111 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:39:30.092312 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086930 5111 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086934 5111 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086938 5111 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086943 5111 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086947 5111 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086951 5111 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086955 5111 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086959 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086963 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086968 5111 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086972 5111 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086976 5111 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086980 5111 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.086984 5111 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087843 5111 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087853 5111 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087858 5111 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087863 5111 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087867 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:39:30.092851 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087872 5111 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087876 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087881 5111 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087885 5111 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087890 5111 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087894 5111 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087899 5111 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087903 5111 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087907 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087912 5111 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087916 5111 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087920 5111 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087925 5111 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087930 5111 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087934 5111 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087938 5111 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087943 5111 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087947 5111 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087951 5111 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087956 5111 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:39:30.093367 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087960 5111 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087967 5111 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087973 5111 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087978 5111 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087984 5111 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087988 5111 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087993 5111 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.087997 5111 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088001 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088006 5111 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088014 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088019 5111 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088023 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088027 5111 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088032 5111 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088036 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088040 5111 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088044 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088049 5111 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088053 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:39:30.093941 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088057 5111 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088061 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088066 5111 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088070 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088074 5111 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088078 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088082 5111 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088087 5111 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088091 5111 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088095 5111 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088100 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088104 5111 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088108 5111 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088112 5111 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088116 5111 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088121 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088125 5111 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088130 5111 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088134 5111 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:39:30.094780 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088138 5111 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088143 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088147 5111 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088153 5111 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088157 5111 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088161 5111 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088167 5111 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088171 5111 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088179 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088184 5111 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088191 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088195 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088199 5111 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088203 5111 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088208 5111 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088212 5111 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088216 5111 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088220 5111 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088224 5111 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:39:30.095407 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088229 5111 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088233 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088238 5111 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088242 5111 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088246 5111 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088251 5111 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088255 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088259 5111 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088263 5111 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088267 5111 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088272 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088276 5111 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088280 5111 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088285 5111 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088289 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088293 5111 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088302 5111 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088306 5111 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088311 5111 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:39:30.095968 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088318 5111 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088325 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088331 5111 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.088335 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088484 5111 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088496 5111 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088504 5111 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088512 5111 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088519 5111 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088525 5111 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088561 5111 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088567 5111 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088572 5111 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088576 5111 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088583 5111 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088589 5111 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088594 5111 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088599 5111 flags.go:64] FLAG: --cgroup-root="" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088604 5111 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088609 5111 flags.go:64] FLAG: --client-ca-file="" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088613 5111 flags.go:64] FLAG: --cloud-provider="external" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088619 5111 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088626 5111 flags.go:64] FLAG: --cluster-domain="" Nov 03 19:39:30.096485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088632 5111 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088636 5111 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088642 5111 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088647 5111 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088653 5111 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088658 5111 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088667 5111 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088672 5111 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088677 5111 flags.go:64] FLAG: --contention-profiling="false" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088682 5111 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088687 5111 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088692 5111 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088696 5111 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088705 5111 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088711 5111 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088716 5111 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088721 5111 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088725 5111 flags.go:64] FLAG: --enable-server="true" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088730 5111 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088737 5111 flags.go:64] FLAG: --event-burst="100" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088742 5111 flags.go:64] FLAG: --event-qps="50" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088747 5111 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088752 5111 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088757 5111 flags.go:64] FLAG: --eviction-hard="" Nov 03 19:39:30.097191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088764 5111 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088769 5111 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088774 5111 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088780 5111 flags.go:64] FLAG: --eviction-soft="" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088785 5111 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088790 5111 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088795 5111 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088800 5111 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088805 5111 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088809 5111 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088814 5111 flags.go:64] FLAG: --feature-gates="" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088821 5111 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088825 5111 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088831 5111 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088838 5111 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088844 5111 flags.go:64] FLAG: --healthz-port="10248" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088850 5111 flags.go:64] FLAG: --help="false" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088854 5111 flags.go:64] FLAG: --hostname-override="ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088860 5111 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088865 5111 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088870 5111 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088877 5111 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088884 5111 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 19:39:30.097853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088889 5111 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088894 5111 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088899 5111 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088903 5111 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088908 5111 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088913 5111 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088918 5111 flags.go:64] FLAG: --kube-reserved="" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088923 5111 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088928 5111 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088933 5111 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088939 5111 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088944 5111 flags.go:64] FLAG: --lock-file="" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088948 5111 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088953 5111 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088957 5111 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088967 5111 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088972 5111 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088977 5111 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088981 5111 flags.go:64] FLAG: --logging-format="text" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088986 5111 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088991 5111 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.088997 5111 flags.go:64] FLAG: --manifest-url="" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089001 5111 flags.go:64] FLAG: --manifest-url-header="" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089011 5111 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089016 5111 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 19:39:30.098435 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089022 5111 flags.go:64] FLAG: --max-pods="110" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089027 5111 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089032 5111 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089037 5111 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089042 5111 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089049 5111 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089054 5111 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089060 5111 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089071 5111 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089076 5111 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089081 5111 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089086 5111 flags.go:64] FLAG: --pod-cidr="" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089091 5111 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089095 5111 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089101 5111 flags.go:64] FLAG: --pods-per-core="0" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089106 5111 flags.go:64] FLAG: --port="10250" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089111 5111 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089116 5111 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-041e0bb2fea46f955" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089121 5111 flags.go:64] FLAG: --qos-reserved="" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089126 5111 flags.go:64] FLAG: --read-only-port="10255" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089131 5111 flags.go:64] FLAG: --register-node="true" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089136 5111 flags.go:64] FLAG: --register-with-taints="" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089142 5111 flags.go:64] FLAG: --registry-burst="10" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089147 5111 flags.go:64] FLAG: --registry-qps="5" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089151 5111 flags.go:64] FLAG: --reserved-cpus="" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089157 5111 flags.go:64] FLAG: --reserved-memory="" Nov 03 19:39:30.099089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089163 5111 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089168 5111 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089173 5111 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089178 5111 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089184 5111 flags.go:64] FLAG: --runonce="false" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089189 5111 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089194 5111 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089199 5111 flags.go:64] FLAG: --seccomp-default="false" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089204 5111 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089209 5111 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089215 5111 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089222 5111 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089227 5111 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089232 5111 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089237 5111 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089242 5111 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089246 5111 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089252 5111 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089257 5111 flags.go:64] FLAG: --system-cgroups="" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089261 5111 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089270 5111 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089275 5111 flags.go:64] FLAG: --tls-cert-file="" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089280 5111 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089288 5111 flags.go:64] FLAG: --tls-min-version="" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089292 5111 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 19:39:30.099772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089298 5111 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089302 5111 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089307 5111 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089312 5111 flags.go:64] FLAG: --v="2" Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089319 5111 flags.go:64] FLAG: --version="false" Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089326 5111 flags.go:64] FLAG: --vmodule="" Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089332 5111 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.089338 5111 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089583 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089591 5111 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089597 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089602 5111 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089607 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089612 5111 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089616 5111 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089621 5111 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089625 5111 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089630 5111 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089639 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089644 5111 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089648 5111 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:39:30.100386 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089652 5111 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089656 5111 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089661 5111 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089665 5111 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089670 5111 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089674 5111 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089679 5111 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089683 5111 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089687 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089692 5111 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089696 5111 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089700 5111 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089706 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089711 5111 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089716 5111 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089720 5111 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089725 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089729 5111 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089734 5111 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:39:30.101048 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089738 5111 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089743 5111 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089747 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089752 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089757 5111 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089761 5111 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089766 5111 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089770 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089775 5111 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089779 5111 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089785 5111 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089790 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089794 5111 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089798 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089803 5111 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089807 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089812 5111 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089816 5111 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089820 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:39:30.101707 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089825 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089829 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089836 5111 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089842 5111 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089847 5111 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089852 5111 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089856 5111 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089861 5111 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089866 5111 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089873 5111 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089879 5111 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089885 5111 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089890 5111 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089895 5111 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089899 5111 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089904 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089908 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089913 5111 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089917 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:39:30.102364 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089921 5111 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089925 5111 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089929 5111 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089935 5111 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089944 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089949 5111 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089954 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089958 5111 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089962 5111 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089966 5111 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089971 5111 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089975 5111 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089979 5111 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089983 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089988 5111 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089992 5111 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.089997 5111 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090001 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090005 5111 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090010 5111 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:39:30.103253 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090014 5111 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:39:30.104205 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090019 5111 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:39:30.104205 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090024 5111 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:39:30.104205 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090028 5111 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:39:30.104205 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090033 5111 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:39:30.104205 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090037 5111 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:39:30.104205 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090041 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:39:30.104205 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090046 5111 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:39:30.104205 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090050 5111 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:39:30.104205 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090055 5111 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:39:30.104205 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090059 5111 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:39:30.104205 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090063 5111 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:39:30.104205 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090068 5111 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:39:30.104205 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090072 5111 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:39:30.104205 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090076 5111 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:39:30.104205 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090081 5111 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:39:30.104205 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.090129 5111 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090194 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090201 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090206 5111 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090211 5111 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090216 5111 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090220 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090224 5111 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090229 5111 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090233 5111 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090239 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090244 5111 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090248 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090253 5111 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090258 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090262 5111 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090266 5111 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090271 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090279 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:39:30.104768 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090284 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090290 5111 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090297 5111 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090303 5111 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090309 5111 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090313 5111 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090318 5111 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090323 5111 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090330 5111 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090336 5111 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090341 5111 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090347 5111 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090351 5111 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090356 5111 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090362 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090366 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090371 5111 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090375 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090380 5111 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:39:30.105351 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090384 5111 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090388 5111 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090392 5111 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090397 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090403 5111 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090407 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090411 5111 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090415 5111 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090420 5111 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090424 5111 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090428 5111 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090432 5111 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090437 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090442 5111 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090447 5111 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090451 5111 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090455 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090460 5111 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090464 5111 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090469 5111 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:39:30.105868 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090473 5111 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090477 5111 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090481 5111 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090485 5111 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090490 5111 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090494 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090499 5111 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090508 5111 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090512 5111 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090516 5111 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090521 5111 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090544 5111 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090549 5111 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090553 5111 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090558 5111 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090562 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090569 5111 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090574 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090578 5111 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090583 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:39:30.106346 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090587 5111 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090592 5111 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090596 5111 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090600 5111 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090605 5111 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090609 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090615 5111 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090620 5111 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090624 5111 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090629 5111 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090633 5111 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090637 5111 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090642 5111 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090646 5111 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090650 5111 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090655 5111 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090659 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090663 5111 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090668 5111 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:39:30.106916 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090674 5111 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090678 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090683 5111 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090687 5111 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090692 5111 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090696 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090700 5111 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090705 5111 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090710 5111 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.090718 5111 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.096399 5111 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.096417 5111 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.096436 5111 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.096441 5111 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096499 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096504 5111 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096508 5111 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096511 5111 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096514 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:39:30.107370 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096517 5111 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096519 5111 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096522 5111 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096525 5111 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096527 5111 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096552 5111 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096556 5111 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096559 5111 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096562 5111 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096567 5111 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096571 5111 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096575 5111 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096578 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096581 5111 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096584 5111 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096587 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096590 5111 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096593 5111 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096595 5111 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:39:30.107862 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096598 5111 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096600 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096603 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096606 5111 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096609 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096611 5111 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096614 5111 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096617 5111 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096619 5111 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096622 5111 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096625 5111 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096627 5111 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096630 5111 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096633 5111 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096636 5111 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096639 5111 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096641 5111 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096644 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096647 5111 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096650 5111 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:39:30.108324 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096652 5111 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096655 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096659 5111 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096661 5111 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096664 5111 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096667 5111 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096670 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096672 5111 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096675 5111 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096677 5111 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096680 5111 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096683 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096685 5111 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096688 5111 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096691 5111 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096693 5111 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096696 5111 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096699 5111 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096702 5111 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096704 5111 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:39:30.108826 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096707 5111 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096710 5111 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096713 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096716 5111 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096719 5111 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096721 5111 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096724 5111 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096727 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096730 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096733 5111 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096735 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096738 5111 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096740 5111 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096743 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096745 5111 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096748 5111 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096751 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096754 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096757 5111 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:39:30.109353 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096759 5111 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096762 5111 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096764 5111 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096767 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096769 5111 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096772 5111 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096774 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096777 5111 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096780 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096783 5111 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096786 5111 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096789 5111 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096792 5111 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096794 5111 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096797 5111 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096800 5111 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096804 5111 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096808 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096812 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:39:30.109850 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096815 5111 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:39:30.110314 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096818 5111 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:39:30.110314 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096820 5111 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:39:30.110314 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096823 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:39:30.110314 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.096851 5111 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:39:30.110314 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096992 5111 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:39:30.110314 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096997 5111 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:39:30.110314 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.096999 5111 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:39:30.110314 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097002 5111 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:39:30.110314 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097005 5111 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:39:30.110314 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097008 5111 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:39:30.110314 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097011 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:39:30.110314 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097014 5111 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:39:30.110314 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097016 5111 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:39:30.110314 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097019 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:39:30.110314 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097022 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097024 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097027 5111 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097029 5111 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097032 5111 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097034 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097037 5111 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097039 5111 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097042 5111 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097044 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097047 5111 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097050 5111 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097052 5111 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097055 5111 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097057 5111 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097061 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097063 5111 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097066 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097068 5111 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:39:30.110762 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097071 5111 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097074 5111 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097077 5111 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097079 5111 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097082 5111 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097084 5111 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097087 5111 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097090 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097092 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097095 5111 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097098 5111 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097100 5111 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097103 5111 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097106 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097108 5111 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097110 5111 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097113 5111 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097115 5111 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097118 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097120 5111 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:39:30.111376 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097123 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097126 5111 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097129 5111 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097132 5111 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097134 5111 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097137 5111 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097139 5111 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097142 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097145 5111 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097148 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097150 5111 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097153 5111 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097155 5111 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097158 5111 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097160 5111 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097163 5111 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097166 5111 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097168 5111 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097171 5111 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097173 5111 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:39:30.112195 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097176 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097179 5111 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097182 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097185 5111 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097187 5111 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097190 5111 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097193 5111 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097195 5111 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097198 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097201 5111 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097203 5111 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097206 5111 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097209 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097211 5111 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097214 5111 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097216 5111 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097219 5111 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097223 5111 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097226 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:39:30.112889 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097229 5111 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097232 5111 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097235 5111 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097238 5111 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097240 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097243 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097247 5111 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097250 5111 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097253 5111 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097256 5111 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097259 5111 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097261 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097264 5111 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097266 5111 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097269 5111 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097271 5111 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097274 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:39:30.113387 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097277 5111 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:39:30.113846 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.097305 5111 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:39:30.113846 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097338 5111 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:39:30.113846 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097342 5111 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:39:30.113846 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097345 5111 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:39:30.113846 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097348 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:39:30.113846 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097351 5111 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:39:30.113846 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097354 5111 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:39:30.113846 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097356 5111 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:39:30.113846 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097359 5111 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:39:30.113846 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097362 5111 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:39:30.113846 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097365 5111 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:39:30.113846 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097368 5111 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:39:30.113846 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097370 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:39:30.113846 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097373 5111 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:39:30.113846 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097376 5111 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:39:30.113846 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097379 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097382 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097384 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097387 5111 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097390 5111 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097392 5111 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097395 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097397 5111 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097400 5111 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097403 5111 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097405 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097408 5111 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097410 5111 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097413 5111 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097415 5111 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097418 5111 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097424 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097427 5111 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097432 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:39:30.114232 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097435 5111 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097437 5111 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097440 5111 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097442 5111 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097445 5111 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097448 5111 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097450 5111 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097453 5111 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097456 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097458 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097461 5111 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097464 5111 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097466 5111 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097470 5111 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097473 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097476 5111 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097479 5111 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097481 5111 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097484 5111 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097486 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:39:30.114766 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097488 5111 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097491 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097494 5111 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097497 5111 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097499 5111 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097520 5111 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097524 5111 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097527 5111 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097543 5111 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097548 5111 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097554 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097559 5111 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097562 5111 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097564 5111 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097568 5111 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097570 5111 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097573 5111 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097576 5111 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097578 5111 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097581 5111 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:39:30.115269 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097584 5111 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097587 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097589 5111 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097592 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097595 5111 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097599 5111 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097602 5111 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097605 5111 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097607 5111 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097610 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097613 5111 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097615 5111 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097618 5111 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097622 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097626 5111 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097630 5111 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097634 5111 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097639 5111 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097643 5111 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:39:30.115822 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097647 5111 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097649 5111 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097652 5111 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097655 5111 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097659 5111 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097662 5111 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097665 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097670 5111 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097673 5111 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097676 5111 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097680 5111 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097683 5111 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097685 5111 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:30.097688 5111 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.097749 5111 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.098763 5111 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.098813 5111 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.098988 5111 server.go:1014] "Starting client certificate rotation" Nov 03 19:39:30.116298 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.099128 5111 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:39:30.116754 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.099176 5111 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:33:50 +0000 UTC" deadline="2028-05-07 02:39:40.048975776 +0000 UTC" Nov 03 19:39:30.116754 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.099216 5111 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13207h0m9.94976212s" Nov 03 19:39:30.116754 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.099460 5111 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:39:30.116754 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.100542 5111 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:39:30.116754 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.101692 5111 log.go:25] "Validated CRI v1 runtime API" Nov 03 19:39:30.116754 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.103986 5111 log.go:25] "Validated CRI v1 image API" Nov 03 19:39:30.116754 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.104772 5111 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 19:39:30.116754 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.106847 5111 fs.go:135] Filesystem UUIDs: map[53ee6b7b-c470-46b2-99b5-818e050ccbd0:/dev/nvme0n1p3 78352053-ef6e-48e0-ab23-8945513478cc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 19:39:30.116994 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.106870 5111 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0657357d1926ac615b5c46eed0196a7b0015b1c6842d7c136771bd5a6fddebfe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0657357d1926ac615b5c46eed0196a7b0015b1c6842d7c136771bd5a6fddebfe/userdata/shm major:0 minor:302 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bfcbddc6f3264cec0e1bda6f9f7f985c0c4f55ae5c1f60aac594851ca3192da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bfcbddc6f3264cec0e1bda6f9f7f985c0c4f55ae5c1f60aac594851ca3192da/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/301dc967e308174461e05be646b0c914b2b4be4d8cf4fb6146577310768f20fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/301dc967e308174461e05be646b0c914b2b4be4d8cf4fb6146577310768f20fb/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/334a4a410b86b9268850049fee7f11d2c1f8958909cc2287bbba5788176f610c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/334a4a410b86b9268850049fee7f11d2c1f8958909cc2287bbba5788176f610c/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46c0a52de1d4ee71e4df2bf1446e536a5c180b6ef54aa5fd211de35d28bc33bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46c0a52de1d4ee71e4df2bf1446e536a5c180b6ef54aa5fd211de35d28bc33bb/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/537f6657fe36c1243633fe52bb29c5601741504015ba8f1347d85850aaa3bda2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/537f6657fe36c1243633fe52bb29c5601741504015ba8f1347d85850aaa3bda2/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58179311e88b6bacd355fcb0851db9a643bc8ac75da54726b904130e3ad6f754/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58179311e88b6bacd355fcb0851db9a643bc8ac75da54726b904130e3ad6f754/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a8d093e45f6dd57c6ae8973495db5795a1a62dded9a37a264bc8eb64d3b0ee7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a8d093e45f6dd57c6ae8973495db5795a1a62dded9a37a264bc8eb64d3b0ee7/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5bedb955f7c772917b34a2abd401baf9ac3d269821138b52552337fbb48b3ec6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5bedb955f7c772917b34a2abd401baf9ac3d269821138b52552337fbb48b3ec6/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e228e40d159d59cf446c8b08af554e1ad50de74c5568e9ce9e3b42ff3ddb0ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e228e40d159d59cf446c8b08af554e1ad50de74c5568e9ce9e3b42ff3ddb0ee/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65a0bf97dec0a5ef6c8f54ba42f048c27928b0d6c5c8e54bcc95c485b11c5a0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65a0bf97dec0a5ef6c8f54ba42f048c27928b0d6c5c8e54bcc95c485b11c5a0d/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6aeb4f12d026ba9495f95d11bfe0501426a34494e2186aa7ef6431675e8818a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6aeb4f12d026ba9495f95d11bfe0501426a34494e2186aa7ef6431675e8818a8/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fcfaa3b721fe1f6c6f44e7656bfdd504a700ef4461095947f0ca7a98333d41b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fcfaa3b721fe1f6c6f44e7656bfdd504a700ef4461095947f0ca7a98333d41b/userdata/shm major:0 minor:300 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80c68326a1ace6d59c0d6ba6d6ed8a0ef57a6bb8d2d46952cae8742a62c92f2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80c68326a1ace6d59c0d6ba6d6ed8a0ef57a6bb8d2d46952cae8742a62c92f2a/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81c832b35f800f18e4bd50b3df9a3225e6b10554ef285a4c8222a7a117461f4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81c832b35f800f18e4bd50b3df9a3225e6b10554ef285a4c8222a7a117461f4e/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/929b901e4ac2e85a87003ac7eab25b79fce54e164c68cd1037616184ac92e757/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/929b901e4ac2e85a87003ac7eab25b79fce54e164c68cd1037616184ac92e757/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/961dfe6bc26775e1420659881bf84559180ac73200913856e80d0d3cf3468246/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/961dfe6bc26775e1420659881bf84559180ac73200913856e80d0d3cf3468246/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98c7739fe14600d71e46476a02d21abd6835ff981f203b953d3ed45564bcc735/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98c7739fe14600d71e46476a02d21abd6835ff981f203b953d3ed45564bcc735/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8259f0b1faea7f133349d60ba4dbdc55499f06e5c55e05afbf25da5003d6b1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8259f0b1faea7f133349d60ba4dbdc55499f06e5c55e05afbf25da5003d6b1d/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c717613fd5693f0bdf38ba2177a00d7871e7cc3d3f33a2425911bfb4a5f12f51/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c717613fd5693f0bdf38ba2177a00d7871e7cc3d3f33a2425911bfb4a5f12f51/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d098744600485db73bc999ffce2fe5f74e8d7db95a1fca5ea601428d7183754f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d098744600485db73bc999ffce2fe5f74e8d7db95a1fca5ea601428d7183754f/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/defefae7588236ad1b108973ce917842a3db6cd60a9b836c99c6637dad7e8b09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/defefae7588236ad1b108973ce917842a3db6cd60a9b836c99c6637dad7e8b09/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3f85f4357a1080a52b18f3e46597fe4fc99fbfc0b8ef0c36127d7c2456a032e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3f85f4357a1080a52b18f3e46597fe4fc99fbfc0b8ef0c36127d7c2456a032e/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5a1225dc4af688d6dc6d0122badf6e3d0ae85b37475721f59943833a97eff20/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5a1225dc4af688d6dc6d0122badf6e3d0ae85b37475721f59943833a97eff20/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9294007165b9536e5b6c299bde0b1c55811e564c02ba55ffba4f109e7466882/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9294007165b9536e5b6c299bde0b1c55811e564c02ba55ffba4f109e7466882/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee7021019df65ebe84ae81b9ef39ba9cfa640e3b78dce3c8024caaa69c2f2142/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee7021019df65ebe84ae81b9ef39ba9cfa640e3b78dce3c8024caaa69c2f2142/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/efa6b54ad3a18df0a75d224a3509f5d0dbd59f5dcbd8423f26f49982c8db8267/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/efa6b54ad3a18df0a75d224a3509f5d0dbd59f5dcbd8423f26f49982c8db8267/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04f022dd-949d-40ab-b00c-902e70ef610e/volumes/kubernetes.io~projected/kube-api-access-gr8k2:{mountpoint:/var/lib/kubelet/pods/04f022dd-949d-40ab-b00c-902e70ef610e/volumes/kubernetes.io~projected/kube-api-access-gr8k2 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0813f12f-0d14-47a8-b4b0-76b159b4e845/volumes/kubernetes.io~projected/kube-api-access-v4bgf:{mountpoint:/var/lib/kubelet/pods/0813f12f-0d14-47a8-b4b0-76b159b4e845/volumes/kubernetes.io~projected/kube-api-access-v4bgf major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09d5c2e9-13cd-4859-abbf-218d18877c5a/volumes/kubernetes.io~projected/kube-api-access-s7qbv:{mountpoint:/var/lib/kubelet/pods/09d5c2e9-13cd-4859-abbf-218d18877c5a/volumes/kubernetes.io~projected/kube-api-access-s7qbv major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fbfc914-8c97-4770-9ef6-6e553c4dcc58/volumes/kubernetes.io~projected/kube-api-access-r5j6b:{mountpoint:/var/lib/kubelet/pods/0fbfc914-8c97-4770-9ef6-6e553c4dcc58/volumes/kubernetes.io~projected/kube-api-access-r5j6b major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19cae502-56f5-4ec2-9867-205025a68d83/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/19cae502-56f5-4ec2-9867-205025a68d83/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b0776e1-a654-4e61-ba94-6d81ddfc8877/volumes/kubernetes.io~projected/kube-api-access-fpk6g:{mountpoint:/var/lib/kubelet/pods/1b0776e1-a654-4e61-ba94-6d81ddfc8877/volumes/kubernetes.io~projected/kube-api-access-fpk6g major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2266dada-9604-44f9-b488-511006cee7b2/volumes/kubernetes.io~projected/kube-api-access-2hcnk:{mountpoint:/var/lib/kubelet/pods/2266dada-9604-44f9-b488-511006cee7b2/volumes/kubernetes.io~projected/kube-api-access-2hcnk major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2266dada-9604-44f9-b488-511006cee7b2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/2266dada-9604-44f9-b488-511006cee7b2/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/257528a3-a0f4-4a07-bba0-67561f8ee205/volumes/kubernetes.io~projected/kube-api-access-9fvg7:{mountpoint:/var/lib/kubelet/pods/257528a3-a0f4-4a07-bba0-67561f8ee205/volumes/kubernetes.io~projected/kube-api-access-9fvg7 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29f79e6b-20d5-4ba4-8737-068f0d82b1e6/volumes/kubernetes.io~projected/kube-api-access-f8x2d:{mountpoint:/var/lib/kubelet/pods/29f79e6b-20d5-4ba4-8737-068f0d82b1e6/volumes/kubernetes.io~projected/kube-api-access-f8x2d major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2dcde652-ceea-4a58-9746-eb2087dea702/volumes/kubernetes.io~projected/kube-api-access-jz9vj:{mountpoint:/var/lib/kubelet/pods/2dcde652-ceea-4a58-9746-eb2087dea702/volumes/kubernetes.io~projected/kube-api-access-jz9vj major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36460aa5-b1f0-4be4-b27b-675c6fecffe1/volumes/kubernetes.io~projected/kube-api-access-567td:{mountpoint:/var/lib/kubelet/pods/36460aa5-b1f0-4be4-b27b-675c6fecffe1/volumes/kubernetes.io~projected/kube-api-access-567td major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/398cf284-004e-463a-a74e-0193946df8b7/volumes/kubernetes.io~projected/kube-api-access-zl29m:{mountpoint:/var/lib/kubelet/pods/398cf284-004e-463a-a74e-0193946df8b7/volumes/kubernetes.io~projected/kube-api-access-zl29m major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44cfa442-7cca-4e81-af9c-789fb885a12c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/44cfa442-7cca-4e81-af9c-789fb885a12c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44cfa442-7cca-4e81-af9c-789fb885a12c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/44cfa442-7cca-4e81-af9c-789fb885a12c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44cfa442-7cca-4e81-af9c-789fb885a12c/volumes/kubernetes.io~projected/kube-api-access-dgn5q:{mountpoint:/var/lib/kubelet/pods/44cfa442-7cca-4e81-af9c-789fb885a12c/volumes/kubernetes.io~projected/kube-api-access-dgn5q major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4734a1de-ee4e-42ab-b925-1a85f65b7d24/volumes/kubernetes.io~projected/kube-api-access-9hdcb:{mountpoint:/var/lib/kubelet/pods/4734a1de-ee4e-42ab-b925-1a85f65b7d24/volumes/kubernetes.io~projected/kube-api-access-9hdcb major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4734a1de-ee4e-42ab-b925-1a85f65b7d24/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/4734a1de-ee4e-42ab-b925-1a85f65b7d24/volumes/kubernetes.io~secret/cert major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5101a457-b490-4538-a9f6-288014fa9264/volumes/kubernetes.io~projected/kube-api-access-rgc9h:{mountpoint:/var/lib/kubelet/pods/5101a457-b490-4538-a9f6-288014fa9264/volumes/kubernetes.io~projected/kube-api-access-rgc9h major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51c48ea1-6b42-4ec0-ae95-bf81b913e901/volumes/kubernetes.io~projected/kube-api-access-25n72:{mountpoint:/var/lib/kubelet/pods/51c48ea1-6b42-4ec0-ae95-bf81b913e901/volumes/kubernetes.io~projected/kube-api-access-25n72 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69c72eb2-3839-4483-82e8-b7d0bf82c57e/volumes/kubernetes.io~projected/kube-api-access-f4w67:{mountpoint:/var/lib/kubelet/pods/69c72eb2-3839-4483-82e8-b7d0bf82c57e/volumes/kubernetes.io~projected/kube-api-access-f4w67 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69c72eb2-3839-4483-82e8-b7d0bf82c57e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/69c72eb2-3839-4483-82e8-b7d0bf82c57e/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bab2991-93a7-44c8-9071-e3c971b79885/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/6bab2991-93a7-44c8-9071-e3c971b79885/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bab2991-93a7-44c8-9071-e3c971b79885/volumes/kubernetes.io~projected/kube-api-access-c67h2:{mountpoint:/var/lib/kubelet/pods/6bab2991-93a7-44c8-9071-e3c971b79885/volumes/kubernetes.io~projected/kube-api-access-c67h2 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bab2991-93a7-44c8-9071-e3c971b79885/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/6bab2991-93a7-44c8-9071-e3c971b79885/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bab2991-93a7-44c8-9071-e3c971b79885/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/6bab2991-93a7-44c8-9071-e3c971b79885/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6efc8dc6-6157-4518-b5fc-dfa8af652691/volumes/kubernetes.io~projected/kube-api-access-2jz64:{mountpoint:/var/lib/kubelet/pods/6efc8dc6-6157-4518-b5fc-dfa8af652691/volumes/kubernetes.io~projected/kube-api-access-2jz64 major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc/volumes/kubernetes.io~projected/kube-api-access-hwnxx:{mountpoint:/var/lib/kubelet/pods/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc/volumes/kubernetes.io~projected/kube-api-access-hwnxx major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc/volumes/kubernetes.io~secret/signing-key major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73f2218a-88f7-4f26-98b3-2d6403bff02d/volumes/kubernetes.io~projected/kube-api-access-9pp9v:{mountpoint:/var/lib/kubelet/pods/73f2218a-88f7-4f26-98b3-2d6403bff02d/volumes/kubernetes.io~projected/kube-api-access-9pp9v major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73f2218a-88f7-4f26-98b3-2d6403bff02d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/73f2218a-88f7-4f26-98b3-2d6403bff02d/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74a9a94d-e73f-4942-bab6-efbfcb728df2/volumes/kubernetes.io~projected/kube-api-access-mmvzw:{mountpoint:/var/lib/kubelet/pods/74a9a94d-e73f-4942-bab6-efbfcb728df2/volumes/kubernetes.io~projected/kube-api-access-mmvzw major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74a9a94d-e73f-4942-bab6-efbfcb728df2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/74a9a94d-e73f-4942-bab6-efbfcb728df2/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/754b8c74-0816-45c8-92f6-95725095e262/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/754b8c74-0816-45c8-92f6-95725095e262/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/754b8c74-0816-45c8-92f6-95725095e262/volumes/kubernetes.io~projected/kube-api-access-b7rsm:{mountpoint:/var/lib/kubelet/pods/754b8c74-0816-45c8-92f6-95725095e262/volumes/kubernetes.io~projected/kube-api-access-b7rsm major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/754b8c74-0816-45c8-92f6-95725095e262/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/754b8c74-0816-45c8-92f6-95725095e262/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9924fd9e-f7b6-4036-8584-22caa1de39b3/volumes/kubernetes.io~projected/kube-api-access-7gk22:{mountpoint:/var/lib/kubelet/pods/9924fd9e-f7b6-4036-8584-22caa1de39b3/volumes/kubernetes.io~projected/kube-api-access-7gk22 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9924fd9e-f7b6-4036-8584-22caa1de39b3/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/9924fd9e-f7b6-4036-8584-22caa1de39b3/volumes/kubernetes.io~secret/default-certificate major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9924fd9e-f7b6-4036-8584-22caa1de39b3/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/9924fd9e-f7b6-4036-8584-22caa1de39b3/volumes/kubernetes.io~secret/stats-auth major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac78f686-0d5a-4ff1-8a04-101e5794b7d7/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ac78f686-0d5a-4ff1-8a04-101e5794b7d7/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac78f686-0d5a-4ff1-8a04-101e5794b7d7/volumes/kubernetes.io~projected/kube-api-access-g4hfr:{mountpoint:/var/lib/kubelet/pods/ac78f686-0d5a-4ff1-8a04-101e5794b7d7/volumes/kubernetes.io~projected/kube-api-access-g4hfr major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac78f686-0d5a-4ff1-8a04-101e5794b7d7/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ac78f686-0d5a-4ff1-8a04-101e5794b7d7/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac78f686-0d5a-4ff1-8a04-101e5794b7d7/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ac78f686-0d5a-4ff1-8a04-101e5794b7d7/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467/volumes/kubernetes.io~projected/kube-api-access-7zcvd:{mountpoint:/var/lib/kubelet/pods/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467/volumes/kubernetes.io~projected/kube-api-access-7zcvd major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca34d416-e238-48cb-a507-953f78fead22/volumes/kubernetes.io~projected/kube-api-access-4qx5n:{mountpoint:/var/lib/kubelet/pods/ca34d416-e238-48cb-a507-953f78fead22/volumes/kubernetes.io~projected/kube-api-access-4qx5n major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca34d416-e238-48cb-a507-953f78fead22/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls:{mountpoint:/var/lib/kubelet/pods/ca34d416-e238-48cb-a507-953f78fead22/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d134b335-9e0e-41c9-a676-566a6ef55f5a/volumes/kubernetes.io~projected/kube-api-access-8hnsk:{mountpoint:/var/lib/kubelet/pods/d134b335-9e0e-41c9-a676-566a6ef55f5a/volumes/kubernetes.io~projected/kube-api-access-8hnsk major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041/volumes/kubernetes.io~projected/kube-api-access-dph2k:{mountpoint:/var/lib/kubelet/pods/d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041/volumes/kubernetes.io~projected/kube-api-access-dph2k major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d85c1bad-d10e-4276-9471-0fbe78bb67b3/volumes/kubernetes.io~projected/kube-api-access-k22vq:{mountpoint:/var/lib/kubelet/pods/d85c1bad-d10e-4276-9471-0fbe78bb67b3/volumes/kubernetes.io~projected/kube-api-access-k22vq major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9e5c9ec-1940-4cb2-8c55-12da74cfc620/volumes/kubernetes.io~projected/kube-api-access-dg6c5:{mountpoint:/var/lib/kubelet/pods/d9e5c9ec-1940-4cb2-8c55-12da74cfc620/volumes/kubernetes.io~projected/kube-api-access-dg6c5 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e90fc007-32bc-43c5-99c6-7dbb3f9753d7/volumes/kubernetes.io~projected/kube-api-access-57r2f:{mountpoint:/var/lib/kubelet/pods/e90fc007-32bc-43c5-99c6-7dbb3f9753d7/volumes/kubernetes.io~projected/kube-api-access-57r2f major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edae5b46-8158-4dc8-91c9-06ed65126db9/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/edae5b46-8158-4dc8-91c9-06ed65126db9/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:270 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/780c4d1d4c02db2471274d555d75f0326fd780ee3bebf843c8e0b0488494b3cf/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e3e05dbaf6eb412844448bde2468e3e99f25f7a20ac61d170289001983d30c4f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/dab76a249f147359960e4cd59811ee68ba53d0aa7ae9b218931dbb6e347296ab/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/9b1f3f7e28c6832412b45978e50f04d9ad690caea97b1b64caee322ea2e5f22a/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/33c936b22c3884fdea9904a5ccd0c2bc836feef06d7a84c9a8499ba0658ba466/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/cc4987cd181a79e9c402bd10cb5d1f2568c66a2bc6315b750b451a3667573efa/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3947bf98dbb3bebfe091c6547f8ff1a79d2e274983805d27700bac0291c7c6df/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3c4aea207e9b054e35357ddc540d19455d8d7b1d6dad813fccc122cc91a6824e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/078c9d0981efe017c19784d754ecc59a7a189040ce456043929cc1930dcf824a/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/901f9f41f469ee12aad696a62de95ace6c6515b7a311ecf2b66f4a19602fb97e/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/2706b2435cf12de43983ae317cdd7eec08d5d722778ec36ff47ca444d97c813d/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/e02a9e35fdcfa8b59de85ab1274723d88a856323978b2440008dc7363a9a856a/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/d96e08df3d82040d292698f7fab9d44d616ac9c9407e152d45c278fb691debdf/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/8dfef85970c5ca6a29eca85650e00d5ba53cc659ce75bc4f947418ffaf11ed04/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/9095ad594ae982d9f95c227353060d747c1c0fdb946c6805b43d5cb88e30b0ac/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d556175b8f091b59189aac1c800f6aa143862d22ac982932e8bedaaeb54d01c5/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/af24da72e6d7d92be7d17d0764a62112dbcaa803a8ce1cb17b336c93adbf689c/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/9c31d560e127352cefc3d3aa255f8bd19bcbf5168cdde1127853780ef3ce53f8/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/792ff0d26ec47fd92c0ac2c92e804e7c5ecd3cc9ab57b087b073c1ce4b6d852c/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/dc5bb12af7c5a8de1b5546c091e994d1f03b284df916412046f9fb3484ca019d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/a156a16b0aa063a9d3ae5b8957bd9c23fa07f16e434933411ea1a1128505a8f5/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/55b9561e9992876a01a513ae6dcbb7e48040e30331808b0573ad3f90cb431fee/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/c6d834acce70777dcadb2ee9d7437284503aa9f8def181a0a605e6cb29bba900/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/8c8fdf5e80e74bbb6fab53997a1931f805dddac6f5be7f7fc9600c1d9bb58e82/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/f85d90d8bac753802a15ce75ca521abfc686e533f8a41163c4a4e34a3e33860a/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/00a6acb46e4b4e8c21f95ef56d1812ad48049a378c67ed26939057d365a89dc6/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/f03471499793f8dfef5ae297bf0463ea30977ae8d48fccba930596598e48e806/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/1ae376f72c253b96a49f003b170e96c20783cdd4d52e5844d72c395b66c57540/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/aa0d798caef1b53f64a3ebdf33a0fa32f0e057ef1fda4892fa554742a28c4d1b/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/895f66b8d182e588190477f0f69609ba11312374b7f750afad014ccf087140b8/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/b5794cf30a61504e8bada43d03ba5502d3c81fa1db96188936931842512af940/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/15ef74c47293c0beccc56ed290804ee54d8dc81fccf55afae505c4bf69594949/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/904febead4281b0acf1ca67c03dd714dd46cc651ff1c1ab74117cf0703002c9a/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/56b81e6e79757810f503352830f0db513737793789da90ddcb13048b583c94d6/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/15edd29bb4dc891cbcdd8a702803ad7f9df26ff7432a2524e03c871f27ff47ad/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/f45a18b39263d898242fddc5825fe17628e78ed30ab6405ad68f0b7e493215e9/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/0561a566e74cb938682aff5aec6c46dc56398c2a1f65e0bde939ea9f10245a2b/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/a380f32807d8de9d25f9bc0ed3e1b369daa3cff7f27149fa51c7ee6a9eed846a/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/6aadaf9843a484e28b44fb9d715c1af779056715f86038126ce820e0142dfb58/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/34b851fa59d4231195fbc9e82c184038ffa51c38a4a9be09e3d6422d3424137a/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/a4dcd3779eb4a94b2aad9b42133ae516e8f69bf85e5cd074f5c4b41b1115a7ea/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/08faf0e0d7e75c9bd2418e11dae3f535e171d6dd701ec38696f3b91353d71613/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/deb77fd55d8c7a60083acfa15e4c72bc27fe7529f4dcfbab932d75f1e8560a4c/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/97b84f80c28bf45a18dd803ffd332c4d0be2955d24068c35048d91bf1f38d03d/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/33cbb82034c395aea36c0bcc5d061f352723eb613d85dc7fd8b9c99731e16be0/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/d1f2cd95e91f5b24c280aa0fb697fb823fce48dc36585a5472a43de3a3a0961f/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/7dce679ffcfa8afae742ce9af9cbfc011e88b3b4258cebb7d7e8941e0ebb2dc4/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/ed5ce498e7d6666703e1d94f8161df282ac816b166c054d16e9480133e687fa2/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/210959c5dd90671fbaef90cb2cd496ec7d5b3545ac8042fc0613d79cf78c6222/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/a13ab469e7a9b3f2dd9e6478c3de7383c6fe07f0d33414aa1faaad12991427ef/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/b1280cb71077da28636553026b3cbb02528290853b19dbf0d1255084f892c55d/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/5c392e8315d446e6f9966f1df7139abf8c4897142da4a17731b02d7588862483/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/58f12e348940d258dfdef8fbafcc383bd8af8f2729e0a7ab83f25929a48e59d7/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/efa9649b6fa8fc6cb755e34e21dee3a487d3c2d15361d820ccf4394027695ead/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/b1c22dfb1e408a8d14d73858e64ef2bb4c3d1f9dbb70d58c32d3f02dbb28a39e/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/0964339f164fd8201769c3c723c29a9e82091dde298dd4cf47aae752fedfd5ca/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/fa2a9c693f1a860a27ef3540624b4ba6649e482ee165355f7354fce7f50251e2/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/f3b5311487ca6b3fea0dffd75b50b03b7d9200e14975bd49d761b5c7c40e8ff3/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/7d0eab19ebb39629bfb65ef5aaabc2ff1b5cfc042373845e489d139da220f5b2/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/ff92db52fbcbd19fce51009117d305959636c2df3978ad99e2ca346cb399be10/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/66fd8da84ce4164bedd36cd5a9bc75f2881e3b44ed9429d625ff8ecb1f0b911e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/2acac74750d0edb42053af3638db9496129fe08a79c6fc6febe1ea56df7dfc85/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e0947ce4fceb66508004432498539c00864fcb37fdb9b2ffbcf603e5f61c3026/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5e723e6d8d02f185b7367d97b1729b4547224dbbf8af89646ffe47bda713bb00/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 19:39:30.118649 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.118247 5111 manager.go:217] Machine: {Timestamp:2026-11-03 19:39:30.11779167 +0000 UTC m=+0.080170101 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3082061 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f66f32789b18489905980b3f146d2 SystemUUID:ec2f66f3-2789-b184-8990-5980b3f146d2 BootID:237797ec-d6e1-4bfd-a68a-bae61361dd56 Filesystems:[{Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/754b8c74-0816-45c8-92f6-95725095e262/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6bab2991-93a7-44c8-9071-e3c971b79885/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6bab2991-93a7-44c8-9071-e3c971b79885/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44cfa442-7cca-4e81-af9c-789fb885a12c/volumes/kubernetes.io~projected/kube-api-access-dgn5q DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51c48ea1-6b42-4ec0-ae95-bf81b913e901/volumes/kubernetes.io~projected/kube-api-access-25n72 DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ac78f686-0d5a-4ff1-8a04-101e5794b7d7/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/537f6657fe36c1243633fe52bb29c5601741504015ba8f1347d85850aaa3bda2/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36460aa5-b1f0-4be4-b27b-675c6fecffe1/volumes/kubernetes.io~projected/kube-api-access-567td DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d098744600485db73bc999ffce2fe5f74e8d7db95a1fca5ea601428d7183754f/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bfcbddc6f3264cec0e1bda6f9f7f985c0c4f55ae5c1f60aac594851ca3192da/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/edae5b46-8158-4dc8-91c9-06ed65126db9/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:270 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8259f0b1faea7f133349d60ba4dbdc55499f06e5c55e05afbf25da5003d6b1d/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6bab2991-93a7-44c8-9071-e3c971b79885/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6efc8dc6-6157-4518-b5fc-dfa8af652691/volumes/kubernetes.io~projected/kube-api-access-2jz64 DeviceMajor:0 DeviceMinor:337 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0813f12f-0d14-47a8-b4b0-76b159b4e845/volumes/kubernetes.io~projected/kube-api-access-v4bgf DeviceMajor:0 DeviceMinor:338 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74a9a94d-e73f-4942-bab6-efbfcb728df2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5101a457-b490-4538-a9f6-288014fa9264/volumes/kubernetes.io~projected/kube-api-access-rgc9h DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fcfaa3b721fe1f6c6f44e7656bfdd504a700ef4461095947f0ca7a98333d41b/userdata/shm DeviceMajor:0 DeviceMinor:300 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/961dfe6bc26775e1420659881bf84559180ac73200913856e80d0d3cf3468246/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9924fd9e-f7b6-4036-8584-22caa1de39b3/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ac78f686-0d5a-4ff1-8a04-101e5794b7d7/volumes/kubernetes.io~projected/kube-api-access-g4hfr DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc/volumes/kubernetes.io~projected/kube-api-access-hwnxx DeviceMajor:0 DeviceMinor:327 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58179311e88b6bacd355fcb0851db9a643bc8ac75da54726b904130e3ad6f754/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d134b335-9e0e-41c9-a676-566a6ef55f5a/volumes/kubernetes.io~projected/kube-api-access-8hnsk DeviceMajor:0 DeviceMinor:341 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/257528a3-a0f4-4a07-bba0-67561f8ee205/volumes/kubernetes.io~projected/kube-api-access-9fvg7 DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6bab2991-93a7-44c8-9071-e3c971b79885/volumes/kubernetes.io~projected/kube-api-access-c67h2 DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b0776e1-a654-4e61-ba94-6d81ddfc8877/volumes/kubernetes.io~projected/kube-api-access-fpk6g DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee7021019df65ebe84ae81b9ef39ba9cfa640e3b78dce3c8024caaa69c2f2142/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac78f686-0d5a-4ff1-8a04-101e5794b7d7/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d85c1bad-d10e-4276-9471-0fbe78bb67b3/volumes/kubernetes.io~projected/kube-api-access-k22vq DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2266dada-9604-44f9-b488-511006cee7b2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ac78f686-0d5a-4ff1-8a04-101e5794b7d7/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/defefae7588236ad1b108973ce917842a3db6cd60a9b836c99c6637dad7e8b09/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e90fc007-32bc-43c5-99c6-7dbb3f9753d7/volumes/kubernetes.io~projected/kube-api-access-57r2f DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/754b8c74-0816-45c8-92f6-95725095e262/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3f85f4357a1080a52b18f3e46597fe4fc99fbfc0b8ef0c36127d7c2456a032e/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/69c72eb2-3839-4483-82e8-b7d0bf82c57e/volumes/kubernetes.io~projected/kube-api-access-f4w67 DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6aeb4f12d026ba9495f95d11bfe0501426a34494e2186aa7ef6431675e8818a8/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44cfa442-7cca-4e81-af9c-789fb885a12c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2dcde652-ceea-4a58-9746-eb2087dea702/volumes/kubernetes.io~projected/kube-api-access-jz9vj DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46c0a52de1d4ee71e4df2bf1446e536a5c180b6ef54aa5fd211de35d28bc33bb/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2266dada-9604-44f9-b488-511006cee7b2/volumes/kubernetes.io~projected/kube-api-access-2hcnk DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/73f2218a-88f7-4f26-98b3-2d6403bff02d/volumes/kubernetes.io~projected/kube-api-access-9pp9v DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/301dc967e308174461e05be646b0c914b2b4be4d8cf4fb6146577310768f20fb/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/44cfa442-7cca-4e81-af9c-789fb885a12c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5a1225dc4af688d6dc6d0122badf6e3d0ae85b37475721f59943833a97eff20/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/929b901e4ac2e85a87003ac7eab25b79fce54e164c68cd1037616184ac92e757/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a8d093e45f6dd57c6ae8973495db5795a1a62dded9a37a264bc8eb64d3b0ee7/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19cae502-56f5-4ec2-9867-205025a68d83/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9294007165b9536e5b6c299bde0b1c55811e564c02ba55ffba4f109e7466882/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80c68326a1ace6d59c0d6ba6d6ed8a0ef57a6bb8d2d46952cae8742a62c92f2a/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/efa6b54ad3a18df0a75d224a3509f5d0dbd59f5dcbd8423f26f49982c8db8267/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/754b8c74-0816-45c8-92f6-95725095e262/volumes/kubernetes.io~projected/kube-api-access-b7rsm DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/29f79e6b-20d5-4ba4-8737-068f0d82b1e6/volumes/kubernetes.io~projected/kube-api-access-f8x2d DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65a0bf97dec0a5ef6c8f54ba42f048c27928b0d6c5c8e54bcc95c485b11c5a0d/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/398cf284-004e-463a-a74e-0193946df8b7/volumes/kubernetes.io~projected/kube-api-access-zl29m DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca34d416-e238-48cb-a507-953f78fead22/volumes/kubernetes.io~projected/kube-api-access-4qx5n DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca34d416-e238-48cb-a507-953f78fead22/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls DeviceMajor:0 DeviceMinor:368 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c717613fd5693f0bdf38ba2177a00d7871e7cc3d3f33a2425911bfb4a5f12f51/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4734a1de-ee4e-42ab-b925-1a85f65b7d24/volumes/kubernetes.io~projected/kube-api-access-9hdcb DeviceMajor:0 DeviceMinor:240 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9e5c9ec-1940-4cb2-8c55-12da74cfc620/volumes/kubernetes.io~projected/kube-api-access-dg6c5 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041/volumes/kubernetes.io~projected/kube-api-access-dph2k DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4734a1de-ee4e-42ab-b925-1a85f65b7d24/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:238 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/334a4a410b86b9268850049fee7f11d2c1f8958909cc2287bbba5788176f610c/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98c7739fe14600d71e46476a02d21abd6835ff981f203b953d3ed45564bcc735/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/04f022dd-949d-40ab-b00c-902e70ef610e/volumes/kubernetes.io~projected/kube-api-access-gr8k2 DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fbfc914-8c97-4770-9ef6-6e553c4dcc58/volumes/kubernetes.io~projected/kube-api-access-r5j6b DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467/volumes/kubernetes.io~projected/kube-api-access-7zcvd DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09d5c2e9-13cd-4859-abbf-218d18877c5a/volumes/kubernetes.io~projected/kube-api-access-s7qbv DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74a9a94d-e73f-4942-bab6-efbfcb728df2/volumes/kubernetes.io~projected/kube-api-access-mmvzw DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81c832b35f800f18e4bd50b3df9a3225e6b10554ef285a4c8222a7a117461f4e/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9924fd9e-f7b6-4036-8584-22caa1de39b3/volumes/kubernetes.io~projected/kube-api-access-7gk22 DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:322 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5bedb955f7c772917b34a2abd401baf9ac3d269821138b52552337fbb48b3ec6/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/73f2218a-88f7-4f26-98b3-2d6403bff02d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/69c72eb2-3839-4483-82e8-b7d0bf82c57e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e228e40d159d59cf446c8b08af554e1ad50de74c5568e9ce9e3b42ff3ddb0ee/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9924fd9e-f7b6-4036-8584-22caa1de39b3/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0657357d1926ac615b5c46eed0196a7b0015b1c6842d7c136771bd5a6fddebfe/userdata/shm DeviceMajor:0 DeviceMinor:302 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0657357d1926ac6 MacAddress:da:2a:07:c5:95:51 Speed:10000 Mtu:8901} {Name:301dc967e308174 MacAddress:a2:df:16:63:fb:71 Speed:10000 Mtu:8901} {Name:334a4a410b86b92 MacAddress:32:8b:ff:1a:0c:c6 Speed:10000 Mtu:8901} {Name:537f6657fe36c12 MacAddress:42:40:23:05:68:1a Speed:10000 Mtu:8901} {Name:58179311e88b6ba MacAddress:7a:8f:8b:2d:db:c3 Speed:10000 Mtu:8901} {Name:5a8d093e45f6dd5 MacAddress:0e:49:f5:d6:00:6b Speed:10000 Mtu:8901} {Name:5e228e40d159d59 MacAddress:56:46:81:22:2f:59 Speed:10000 Mtu:8901} {Name:65a0bf97dec0a5e MacAddress:5e:cb:12:f6:3c:19 Speed:10000 Mtu:8901} {Name:6aeb4f12d026ba9 MacAddress:7e:9d:d4:80:31:e6 Speed:10000 Mtu:8901} {Name:6fcfaa3b721fe1f MacAddress:5e:0e:91:d9:bc:0d Speed:10000 Mtu:8901} {Name:80c68326a1ace6d MacAddress:6e:13:ba:99:ca:14 Speed:10000 Mtu:8901} {Name:81c832b35f800f1 MacAddress:2a:94:de:ed:01:45 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fd:4d:29:d1 Speed:0 Mtu:9001} {Name:br-int MacAddress:da:4d:46:62:e6:1c Speed:0 Mtu:8901} {Name:c717613fd5693f0 MacAddress:56:38:65:0a:ca:9a Speed:10000 Mtu:8901} {Name:defefae7588236a MacAddress:72:28:4e:98:8b:ec Speed:10000 Mtu:8901} {Name:e5a1225dc4af688 MacAddress:96:c1:de:d2:39:eb Speed:10000 Mtu:8901} {Name:ee7021019df65eb MacAddress:c2:85:8c:14:e9:e9 Speed:10000 Mtu:8901} {Name:efa6b54ad3a18df MacAddress:56:52:a0:2c:df:7a Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:fd:4d:29:d1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:b6:82:d6:ed:62:f4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:96:07:4a:0d:92:d6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 19:39:30.118865 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.118653 5111 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 19:39:30.118865 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.118700 5111 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 19:39:30.119059 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.119030 5111 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 19:39:30.119245 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.119064 5111 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-178.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 19:39:30.119288 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.119249 5111 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 19:39:30.119288 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.119261 5111 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 19:39:30.119288 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.119274 5111 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:39:30.119368 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.119291 5111 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:39:30.119414 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.119405 5111 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 19:39:30.119642 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.119631 5111 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 19:39:30.119796 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.119786 5111 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 19:39:30.119839 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.119830 5111 kubelet.go:493] "Attempting to sync node with API server" Nov 03 19:39:30.119839 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.119846 5111 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 19:39:30.119905 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.119861 5111 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 19:39:30.119905 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.119874 5111 kubelet.go:405] "Adding apiserver pod source" Nov 03 19:39:30.119905 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.119885 5111 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 19:39:30.120790 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.120776 5111 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 19:39:30.120862 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.120791 5111 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:39:30.121348 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.121325 5111 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 19:39:30.123571 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.123554 5111 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 19:39:30.123571 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.123568 5111 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:39:30.123678 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.123670 5111 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 19:39:30.123863 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.123851 5111 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 19:39:30.124139 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.123892 5111 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 19:39:30.124139 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.124062 5111 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 19:39:30.124139 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.124081 5111 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 19:39:30.124139 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.124090 5111 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 19:39:30.124139 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.124099 5111 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 19:39:30.124139 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.124109 5111 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 19:39:30.124139 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.124119 5111 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 19:39:30.124139 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.124128 5111 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 19:39:30.124139 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.124137 5111 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 19:39:30.124422 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.124148 5111 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 19:39:30.124422 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.124158 5111 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 19:39:30.124422 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.124171 5111 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 19:39:30.124422 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.124185 5111 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 19:39:30.124422 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.124217 5111 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 19:39:30.124422 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.124226 5111 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 19:39:30.130710 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.128153 5111 server.go:1263] "Started kubelet" Nov 03 19:39:30.130710 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.128650 5111 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 19:39:30.130710 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.128943 5111 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 19:39:30.130710 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.129018 5111 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 19:39:30.130710 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.129428 5111 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 19:39:30.128729 ip-10-0-134-178 systemd[1]: Started Kubernetes Kubelet. Nov 03 19:39:30.133398 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.133268 5111 server.go:317] "Adding debug handlers to kubelet server" Nov 03 19:39:30.134467 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.134450 5111 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.134759 5111 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.134766 5111 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.134809 5111 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:30.134844 5111 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-178.us-east-2.compute.internal\" not found" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.134854 5111 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.134775 5111 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.137402 5111 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:33:51 +0000 UTC" deadline="2028-07-27 19:28:11.715165154 +0000 UTC" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.137422 5111 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15167h48m41.577746138s" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.138731 5111 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.139242 5111 factory.go:55] Registering systemd factory Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.139258 5111 factory.go:223] Registration of the systemd container factory successfully Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140169 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09d5c2e9-13cd-4859-abbf-218d18877c5a" volumeName="kubernetes.io/projected/09d5c2e9-13cd-4859-abbf-218d18877c5a-kube-api-access-s7qbv" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140213 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69c72eb2-3839-4483-82e8-b7d0bf82c57e" volumeName="kubernetes.io/secret/69c72eb2-3839-4483-82e8-b7d0bf82c57e-serving-cert" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140227 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b0776e1-a654-4e61-ba94-6d81ddfc8877" volumeName="kubernetes.io/projected/1b0776e1-a654-4e61-ba94-6d81ddfc8877-kube-api-access-fpk6g" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140238 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2266dada-9604-44f9-b488-511006cee7b2" volumeName="kubernetes.io/empty-dir/2266dada-9604-44f9-b488-511006cee7b2-snapshots" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140250 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f79e6b-20d5-4ba4-8737-068f0d82b1e6" volumeName="kubernetes.io/empty-dir/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-tmp-dir" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140261 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44cfa442-7cca-4e81-af9c-789fb885a12c" volumeName="kubernetes.io/projected/44cfa442-7cca-4e81-af9c-789fb885a12c-kube-api-access-dgn5q" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140271 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4734a1de-ee4e-42ab-b925-1a85f65b7d24" volumeName="kubernetes.io/projected/4734a1de-ee4e-42ab-b925-1a85f65b7d24-kube-api-access-9hdcb" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140282 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="754b8c74-0816-45c8-92f6-95725095e262" volumeName="kubernetes.io/projected/754b8c74-0816-45c8-92f6-95725095e262-kube-api-access-b7rsm" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140296 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9e5c9ec-1940-4cb2-8c55-12da74cfc620" volumeName="kubernetes.io/projected/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-kube-api-access-dg6c5" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140308 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04f022dd-949d-40ab-b00c-902e70ef610e" volumeName="kubernetes.io/projected/04f022dd-949d-40ab-b00c-902e70ef610e-kube-api-access-gr8k2" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140319 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f79e6b-20d5-4ba4-8737-068f0d82b1e6" volumeName="kubernetes.io/projected/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-kube-api-access-f8x2d" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140330 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69c72eb2-3839-4483-82e8-b7d0bf82c57e" volumeName="kubernetes.io/projected/69c72eb2-3839-4483-82e8-b7d0bf82c57e-kube-api-access-f4w67" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140340 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fd367f3-a564-4922-8d7b-2d4b84eaa1bc" volumeName="kubernetes.io/secret/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-signing-key" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140351 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="754b8c74-0816-45c8-92f6-95725095e262" volumeName="kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-ovnkube-script-lib" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140365 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac78f686-0d5a-4ff1-8a04-101e5794b7d7" volumeName="kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-image-registry-private-configuration" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140375 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73f2218a-88f7-4f26-98b3-2d6403bff02d" volumeName="kubernetes.io/projected/73f2218a-88f7-4f26-98b3-2d6403bff02d-kube-api-access-9pp9v" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140392 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac78f686-0d5a-4ff1-8a04-101e5794b7d7" volumeName="kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-bound-sa-token" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140403 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36460aa5-b1f0-4be4-b27b-675c6fecffe1" volumeName="kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140415 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36460aa5-b1f0-4be4-b27b-675c6fecffe1" volumeName="kubernetes.io/projected/36460aa5-b1f0-4be4-b27b-675c6fecffe1-kube-api-access-567td" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140430 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51c48ea1-6b42-4ec0-ae95-bf81b913e901" volumeName="kubernetes.io/projected/51c48ea1-6b42-4ec0-ae95-bf81b913e901-kube-api-access-25n72" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140444 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="754b8c74-0816-45c8-92f6-95725095e262" volumeName="kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-ovnkube-config" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140455 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9924fd9e-f7b6-4036-8584-22caa1de39b3" volumeName="kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-stats-auth" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140466 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac78f686-0d5a-4ff1-8a04-101e5794b7d7" volumeName="kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-kube-api-access-g4hfr" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140477 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac78f686-0d5a-4ff1-8a04-101e5794b7d7" volumeName="kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-installation-pull-secrets" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140488 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041" volumeName="kubernetes.io/projected/d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041-kube-api-access-dph2k" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140499 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fbfc914-8c97-4770-9ef6-6e553c4dcc58" volumeName="kubernetes.io/projected/0fbfc914-8c97-4770-9ef6-6e553c4dcc58-kube-api-access-r5j6b" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140510 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19cae502-56f5-4ec2-9867-205025a68d83" volumeName="kubernetes.io/configmap/19cae502-56f5-4ec2-9867-205025a68d83-konnectivity-ca" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140521 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5101a457-b490-4538-a9f6-288014fa9264" volumeName="kubernetes.io/empty-dir/5101a457-b490-4538-a9f6-288014fa9264-tmp-dir" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140549 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2266dada-9604-44f9-b488-511006cee7b2" volumeName="kubernetes.io/configmap/2266dada-9604-44f9-b488-511006cee7b2-service-ca-bundle" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140560 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44cfa442-7cca-4e81-af9c-789fb885a12c" volumeName="kubernetes.io/empty-dir/44cfa442-7cca-4e81-af9c-789fb885a12c-tmp" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140579 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73f2218a-88f7-4f26-98b3-2d6403bff02d" volumeName="kubernetes.io/configmap/73f2218a-88f7-4f26-98b3-2d6403bff02d-trusted-ca" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140589 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74a9a94d-e73f-4942-bab6-efbfcb728df2" volumeName="kubernetes.io/configmap/74a9a94d-e73f-4942-bab6-efbfcb728df2-config" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140612 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d134b335-9e0e-41c9-a676-566a6ef55f5a" volumeName="kubernetes.io/empty-dir/d134b335-9e0e-41c9-a676-566a6ef55f5a-data-volume" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140624 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d134b335-9e0e-41c9-a676-566a6ef55f5a" volumeName="kubernetes.io/projected/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-api-access-8hnsk" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140636 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2266dada-9604-44f9-b488-511006cee7b2" volumeName="kubernetes.io/projected/2266dada-9604-44f9-b488-511006cee7b2-kube-api-access-2hcnk" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140647 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2266dada-9604-44f9-b488-511006cee7b2" volumeName="kubernetes.io/secret/2266dada-9604-44f9-b488-511006cee7b2-serving-cert" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140657 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dcde652-ceea-4a58-9746-eb2087dea702" volumeName="kubernetes.io/projected/2dcde652-ceea-4a58-9746-eb2087dea702-kube-api-access-jz9vj" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140669 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bab2991-93a7-44c8-9071-e3c971b79885" volumeName="kubernetes.io/empty-dir/6bab2991-93a7-44c8-9071-e3c971b79885-ca-trust-extracted" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140680 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bab2991-93a7-44c8-9071-e3c971b79885" volumeName="kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-kube-api-access-c67h2" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140690 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fd367f3-a564-4922-8d7b-2d4b84eaa1bc" volumeName="kubernetes.io/projected/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-kube-api-access-hwnxx" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140701 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="754b8c74-0816-45c8-92f6-95725095e262" volumeName="kubernetes.io/secret/754b8c74-0816-45c8-92f6-95725095e262-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140711 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac78f686-0d5a-4ff1-8a04-101e5794b7d7" volumeName="kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-trusted-ca" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140722 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="398cf284-004e-463a-a74e-0193946df8b7" volumeName="kubernetes.io/configmap/398cf284-004e-463a-a74e-0193946df8b7-multus-daemon-config" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140732 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bab2991-93a7-44c8-9071-e3c971b79885" volumeName="kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-bound-sa-token" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140743 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="754b8c74-0816-45c8-92f6-95725095e262" volumeName="kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-env-overrides" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140754 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac78f686-0d5a-4ff1-8a04-101e5794b7d7" volumeName="kubernetes.io/empty-dir/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-ca-trust-extracted" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140764 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9e5c9ec-1940-4cb2-8c55-12da74cfc620" volumeName="kubernetes.io/configmap/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-iptables-alerter-script" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140777 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2266dada-9604-44f9-b488-511006cee7b2" volumeName="kubernetes.io/configmap/2266dada-9604-44f9-b488-511006cee7b2-trusted-ca-bundle" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140788 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="398cf284-004e-463a-a74e-0193946df8b7" volumeName="kubernetes.io/configmap/398cf284-004e-463a-a74e-0193946df8b7-cni-binary-copy" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140798 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69c72eb2-3839-4483-82e8-b7d0bf82c57e" volumeName="kubernetes.io/configmap/69c72eb2-3839-4483-82e8-b7d0bf82c57e-config" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140810 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6efc8dc6-6157-4518-b5fc-dfa8af652691" volumeName="kubernetes.io/projected/6efc8dc6-6157-4518-b5fc-dfa8af652691-kube-api-access-2jz64" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140820 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73f2218a-88f7-4f26-98b3-2d6403bff02d" volumeName="kubernetes.io/configmap/73f2218a-88f7-4f26-98b3-2d6403bff02d-config" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140833 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2503201-686a-4cf9-b5a1-977c87e6b717" volumeName="kubernetes.io/configmap/b2503201-686a-4cf9-b5a1-977c87e6b717-nginx-conf" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140844 5111 factory.go:153] Registering CRI-O factory Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140862 5111 factory.go:223] Registration of the crio container factory successfully Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140912 5111 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140845 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467" volumeName="kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-service-ca-bundle" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140930 5111 factory.go:103] Registering Raw factory Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140931 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467" volumeName="kubernetes.io/projected/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-kube-api-access-7zcvd" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140945 5111 manager.go:1199] Started watching for new ooms in manager Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.140943 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0813f12f-0d14-47a8-b4b0-76b159b4e845" volumeName="kubernetes.io/projected/0813f12f-0d14-47a8-b4b0-76b159b4e845-kube-api-access-v4bgf" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141018 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="257528a3-a0f4-4a07-bba0-67561f8ee205" volumeName="kubernetes.io/projected/257528a3-a0f4-4a07-bba0-67561f8ee205-kube-api-access-9fvg7" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141044 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bab2991-93a7-44c8-9071-e3c971b79885" volumeName="kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-image-registry-private-configuration" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141061 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74a9a94d-e73f-4942-bab6-efbfcb728df2" volumeName="kubernetes.io/projected/74a9a94d-e73f-4942-bab6-efbfcb728df2-kube-api-access-mmvzw" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141073 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74a9a94d-e73f-4942-bab6-efbfcb728df2" volumeName="kubernetes.io/secret/74a9a94d-e73f-4942-bab6-efbfcb728df2-serving-cert" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141085 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9924fd9e-f7b6-4036-8584-22caa1de39b3" volumeName="kubernetes.io/projected/9924fd9e-f7b6-4036-8584-22caa1de39b3-kube-api-access-7gk22" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141101 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9924fd9e-f7b6-4036-8584-22caa1de39b3" volumeName="kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-default-certificate" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141115 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac78f686-0d5a-4ff1-8a04-101e5794b7d7" volumeName="kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-certificates" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141131 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467" volumeName="kubernetes.io/empty-dir/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-archives-path" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141144 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca34d416-e238-48cb-a507-953f78fead22" volumeName="kubernetes.io/projected/ca34d416-e238-48cb-a507-953f78fead22-kube-api-access-4qx5n" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141157 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca34d416-e238-48cb-a507-953f78fead22" volumeName="kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141170 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d85c1bad-d10e-4276-9471-0fbe78bb67b3" volumeName="kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141183 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e90fc007-32bc-43c5-99c6-7dbb3f9753d7" volumeName="kubernetes.io/configmap/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-serviceca" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141197 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edae5b46-8158-4dc8-91c9-06ed65126db9" volumeName="kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141209 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19cae502-56f5-4ec2-9867-205025a68d83" volumeName="kubernetes.io/secret/19cae502-56f5-4ec2-9867-205025a68d83-agent-certs" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141222 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b0776e1-a654-4e61-ba94-6d81ddfc8877" volumeName="kubernetes.io/configmap/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-config" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141235 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2266dada-9604-44f9-b488-511006cee7b2" volumeName="kubernetes.io/empty-dir/2266dada-9604-44f9-b488-511006cee7b2-tmp" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141248 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4734a1de-ee4e-42ab-b925-1a85f65b7d24" volumeName="kubernetes.io/secret/4734a1de-ee4e-42ab-b925-1a85f65b7d24-cert" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141265 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5101a457-b490-4538-a9f6-288014fa9264" volumeName="kubernetes.io/configmap/5101a457-b490-4538-a9f6-288014fa9264-config-volume" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141280 5111 manager.go:319] Starting recovery of all containers Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141279 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bab2991-93a7-44c8-9071-e3c971b79885" volumeName="kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-trusted-ca" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141430 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73f2218a-88f7-4f26-98b3-2d6403bff02d" volumeName="kubernetes.io/secret/73f2218a-88f7-4f26-98b3-2d6403bff02d-serving-cert" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141445 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca34d416-e238-48cb-a507-953f78fead22" volumeName="kubernetes.io/configmap/ca34d416-e238-48cb-a507-953f78fead22-telemetry-config" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141457 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36460aa5-b1f0-4be4-b27b-675c6fecffe1" volumeName="kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cni-binary-copy" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141470 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bab2991-93a7-44c8-9071-e3c971b79885" volumeName="kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-registry-certificates" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141481 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36460aa5-b1f0-4be4-b27b-675c6fecffe1" volumeName="kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141494 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5101a457-b490-4538-a9f6-288014fa9264" volumeName="kubernetes.io/projected/5101a457-b490-4538-a9f6-288014fa9264-kube-api-access-rgc9h" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141506 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fd367f3-a564-4922-8d7b-2d4b84eaa1bc" volumeName="kubernetes.io/configmap/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-signing-cabundle" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141519 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="398cf284-004e-463a-a74e-0193946df8b7" volumeName="kubernetes.io/projected/398cf284-004e-463a-a74e-0193946df8b7-kube-api-access-zl29m" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141620 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44cfa442-7cca-4e81-af9c-789fb885a12c" volumeName="kubernetes.io/empty-dir/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-tuned" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141635 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bab2991-93a7-44c8-9071-e3c971b79885" volumeName="kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-installation-pull-secrets" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141647 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467" volumeName="kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-trusted-ca-bundle" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141659 5111 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e90fc007-32bc-43c5-99c6-7dbb3f9753d7" volumeName="kubernetes.io/projected/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-kube-api-access-57r2f" seLinuxMountContext="" Nov 03 19:39:30.141570 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141671 5111 reconstruct.go:97] "Volume reconstruction finished" Nov 03 19:39:30.146630 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.141678 5111 reconciler.go:29] "Reconciler: start to sync state" Nov 03 19:39:30.146630 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.144428 5111 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 19:39:30.160258 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.160238 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 19:39:30.161640 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.160818 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 19:39:30.166080 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.163653 5111 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:39:30.169852 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.169829 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 19:39:30.169987 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.169967 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 19:39:30.172647 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.172606 5111 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 19:39:30.174254 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.174233 5111 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 19:39:30.174254 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.174255 5111 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 19:39:30.174401 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.174275 5111 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 19:39:30.174401 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:30.174313 5111 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 19:39:30.177852 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.177833 5111 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:39:30.190446 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.190416 5111 generic.go:356] "Generic (PLEG): container finished" podID="36460aa5-b1f0-4be4-b27b-675c6fecffe1" containerID="4dd5fabd891fa76438ef2de301c3e15d52ae5e95f6c7b9ca09bf0f1995e8b47f" exitCode=0 Nov 03 19:39:30.190446 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.190444 5111 generic.go:356] "Generic (PLEG): container finished" podID="36460aa5-b1f0-4be4-b27b-675c6fecffe1" containerID="348f6a0228a555e67e1c61a848d6a87a4d42568f2f75da672741905764ec43a7" exitCode=0 Nov 03 19:39:30.190636 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.190454 5111 generic.go:356] "Generic (PLEG): container finished" podID="36460aa5-b1f0-4be4-b27b-675c6fecffe1" containerID="2cae42d4d47b384bd3c82b237c1af72be88663ad82f48b076bf27d4e9601440d" exitCode=0 Nov 03 19:39:30.190636 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.190463 5111 generic.go:356] "Generic (PLEG): container finished" podID="36460aa5-b1f0-4be4-b27b-675c6fecffe1" containerID="3bfcadb20970ec7bb9d2dfcc4a3674a27bdf23ab0189ab81d06c628190219ee9" exitCode=0 Nov 03 19:39:30.190636 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.190469 5111 generic.go:356] "Generic (PLEG): container finished" podID="36460aa5-b1f0-4be4-b27b-675c6fecffe1" containerID="e6e10c696a9e5ee879102b80645f170a411b1d0283d20677f12313e18e55eb77" exitCode=0 Nov 03 19:39:30.190636 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.190479 5111 generic.go:356] "Generic (PLEG): container finished" podID="36460aa5-b1f0-4be4-b27b-675c6fecffe1" containerID="76daf4917c1d96eb4e4198fbe17526941397a4cf2938c711b49f7224257d93fc" exitCode=0 Nov 03 19:39:30.208129 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.208100 5111 generic.go:356] "Generic (PLEG): container finished" podID="b96c57e6fe4ecb21f4910168c569318a" containerID="a64a5e51e39b7d2468c1684754edff6275bb63b663834b581daf33638ea4b322" exitCode=0 Nov 03 19:39:30.210225 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.210206 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 19:39:30.210338 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.210236 5111 generic.go:356] "Generic (PLEG): container finished" podID="73f2218a-88f7-4f26-98b3-2d6403bff02d" containerID="cc6f5944bc7c8c6ba63cc9c0e8ce8c89768dca1e2ed1b8fc6ea3065c53dfeb5e" exitCode=255 Nov 03 19:39:30.231403 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.231350 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 19:39:30.231702 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.231676 5111 generic.go:356] "Generic (PLEG): container finished" podID="754b8c74-0816-45c8-92f6-95725095e262" containerID="2d5a632d180fdebf9d0372a733e77abeb0f2a16003a3f308ad0bb22eb5edff59" exitCode=1 Nov 03 19:39:30.253300 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.253282 5111 manager.go:324] Recovery completed Nov 03 19:39:30.274436 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:30.274398 5111 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 19:39:30.275462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.275437 5111 cpu_manager.go:226] "Starting" policy="none" Nov 03 19:39:30.275462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.275461 5111 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 19:39:30.275630 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.275484 5111 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 19:39:30.275711 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.275693 5111 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 19:39:30.275766 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.275713 5111 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 19:39:30.275766 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.275736 5111 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 19:39:30.275766 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.275744 5111 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 19:39:30.275766 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.275751 5111 policy_none.go:50] "Start" Nov 03 19:39:30.275932 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.275772 5111 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 19:39:30.275932 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.275785 5111 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 19:39:30.287403 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.287370 5111 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 19:39:30.287403 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.287400 5111 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 19:39:30.287403 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.287409 5111 policy_none.go:44] "Start" Nov 03 19:39:30.292340 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.292310 5111 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 19:39:30.294882 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.294869 5111 manager.go:341] "Starting Device Plugin manager" Nov 03 19:39:30.295350 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:30.294892 5111 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 19:39:30.295350 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.295352 5111 server.go:87] "Starting device plugin registration server" Nov 03 19:39:30.295628 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.295616 5111 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 19:39:30.295698 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.295628 5111 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 19:39:30.295766 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.295745 5111 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 19:39:30.295897 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.295860 5111 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 19:39:30.295946 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.295907 5111 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 19:39:30.295946 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.295915 5111 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 19:39:30.296053 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.295969 5111 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:39:30.295895753Z","UUID":"a0e57155-4027-4a1b-9fbe-7f82e18e7b23","Handler":null,"Name":"","Endpoint":""} Nov 03 19:39:30.297304 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:30.297280 5111 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 19:39:30.297416 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.297401 5111 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 19:39:30.297507 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.297425 5111 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 19:39:30.397194 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.397156 5111 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:39:30.398472 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.398450 5111 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:39:30.398618 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.398485 5111 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:39:30.398618 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.398496 5111 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-178.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:39:30.398618 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.398574 5111 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:30.410981 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.410949 5111 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:30.411150 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.411044 5111 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:30.475084 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.475026 5111 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal"] Nov 03 19:39:30.475753 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.475697 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" event={"ID":"5cf8bc898cbd908afece93ec1ad60d21","Type":"ContainerStarted","Data":"df7ae75ab0d6401a6a7a88804d82484cc0212a4723cdd51e9655983291ce9599"} Nov 03 19:39:30.475878 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.475767 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" event={"ID":"5cf8bc898cbd908afece93ec1ad60d21","Type":"ContainerStarted","Data":"e3f85f4357a1080a52b18f3e46597fe4fc99fbfc0b8ef0c36127d7c2456a032e"} Nov 03 19:39:30.475878 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.475787 5111 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:30.475878 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.475811 5111 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:30.475878 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.475819 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" event={"ID":"b96c57e6fe4ecb21f4910168c569318a","Type":"ContainerStarted","Data":"9083153c1cab1e7b4fa4bb03b8aa6973676185bd2a20dd78ff88bf91d5301648"} Nov 03 19:39:30.475878 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.475841 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" event={"ID":"b96c57e6fe4ecb21f4910168c569318a","Type":"ContainerDied","Data":"a64a5e51e39b7d2468c1684754edff6275bb63b663834b581daf33638ea4b322"} Nov 03 19:39:30.475878 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.475858 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" event={"ID":"b96c57e6fe4ecb21f4910168c569318a","Type":"ContainerStarted","Data":"929b901e4ac2e85a87003ac7eab25b79fce54e164c68cd1037616184ac92e757"} Nov 03 19:39:30.488651 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.488600 5111 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:39:30.488651 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:30.488646 5111 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:30.488949 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.488939 5111 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:39:30.488990 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:30.488962 5111 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:30.545314 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.545284 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b96c57e6fe4ecb21f4910168c569318a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"b96c57e6fe4ecb21f4910168c569318a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:30.545314 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.545314 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cf8bc898cbd908afece93ec1ad60d21-config\") pod \"kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"5cf8bc898cbd908afece93ec1ad60d21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:30.545492 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.545333 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b96c57e6fe4ecb21f4910168c569318a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"b96c57e6fe4ecb21f4910168c569318a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:30.646249 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.646216 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b96c57e6fe4ecb21f4910168c569318a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"b96c57e6fe4ecb21f4910168c569318a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:30.646390 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.646251 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b96c57e6fe4ecb21f4910168c569318a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"b96c57e6fe4ecb21f4910168c569318a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:30.646390 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.646273 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cf8bc898cbd908afece93ec1ad60d21-config\") pod \"kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"5cf8bc898cbd908afece93ec1ad60d21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:30.646390 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.646321 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5cf8bc898cbd908afece93ec1ad60d21-config\") pod \"kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"5cf8bc898cbd908afece93ec1ad60d21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:30.646390 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.646342 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b96c57e6fe4ecb21f4910168c569318a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"b96c57e6fe4ecb21f4910168c569318a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:30.646390 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:30.646382 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b96c57e6fe4ecb21f4910168c569318a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal\" (UID: \"b96c57e6fe4ecb21f4910168c569318a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:31.120087 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.120004 5111 apiserver.go:52] "Watching apiserver" Nov 03 19:39:31.134224 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.134192 5111 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 19:39:31.134953 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.134924 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-sk4zs","openshift-image-registry/image-registry-b48f6466f-rvj5f","openshift-multus/multus-npg2d","openshift-dns/node-resolver-k666m","kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal","openshift-dns/dns-default-zdf6j","kube-system/global-pull-secret-syncer-p4t7z","openshift-kube-storage-version-migrator/migrator-66df5d496b-slmpf","openshift-image-registry/image-registry-85444c64c6-qk7p8","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-txtq2","openshift-console-operator/console-operator-7cd56b8dbc-68j2z","openshift-insights/insights-runtime-extractor-md5d2","openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8","openshift-kube-storage-version-migrator/migrator-66df5d496b-t8rfb","openshift-multus/multus-additional-cni-plugins-w6vlx","kube-system/kas-connection-checker-685895c7c9-knv7j","openshift-multus/network-metrics-daemon-z9svh","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p","openshift-network-diagnostics/network-check-target-wztqw","openshift-service-ca/service-ca-5fffc97bdf-fckg5","openshift-ovn-kubernetes/ovnkube-node-tzzd5","kube-system/kas-connection-checker-685895c7c9-x9cw5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal","openshift-ingress-canary/ingress-canary-ggbpk","openshift-cluster-node-tuning-operator/tuned-sks78","openshift-network-operator/iptables-alerter-rhlgs","openshift-insights/insights-operator-7c4986d449-rs8wk","openshift-ingress/router-default-855cd4cff-prjqj","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck","kube-system/konnectivity-agent-f4x7r","openshift-network-diagnostics/network-check-source-6f78d59f99-f9vq4","openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw","kube-system/kas-connection-checker-685895c7c9-d5qxk","openshift-image-registry/node-ca-pvs99","openshift-insights/periodic-gathering-fn7j5-kgmpq"] Nov 03 19:39:31.136550 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.136518 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:31.138052 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.138031 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.139346 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.139320 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:31.141422 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.141394 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:31.141554 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.141519 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.141722 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.141699 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.141792 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.141741 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 19:39:31.141917 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.141890 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gchmt\"" Nov 03 19:39:31.142001 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.141977 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.142102 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.141982 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.142413 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.142399 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:31.142549 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.142512 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 19:39:31.142799 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.142776 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-sstjk\"" Nov 03 19:39:31.142861 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.142838 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 19:39:31.144074 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.144047 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.145134 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.145111 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 19:39:31.145236 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.145110 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 19:39:31.145236 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.145214 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 19:39:31.145236 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.145225 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 19:39:31.145379 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.145321 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.145379 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.145353 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 19:39:31.145473 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.145441 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.145473 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.145459 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 19:39:31.145593 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.145562 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.145719 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.145700 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.146614 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.146592 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 19:39:31.146698 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.146618 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.146764 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.146748 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mwk25\"" Nov 03 19:39:31.146852 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.146837 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x24rb\"" Nov 03 19:39:31.148896 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.148879 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.151399 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.151384 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.152428 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.152404 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 19:39:31.152489 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.152414 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.152702 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.152683 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 19:39:31.152773 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.152698 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 19:39:31.152773 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.152749 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.152874 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.152800 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.152951 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.152933 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 19:39:31.153004 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.152962 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 19:39:31.153004 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.152965 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.153004 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.152989 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.153004 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.153004 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 19:39:31.153181 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.153008 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.153181 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.153019 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.153181 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.153035 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.153687 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.153673 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:31.154131 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.154115 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.154131 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.154115 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.154232 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.154140 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.154400 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.154382 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 19:39:31.154482 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.154416 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-869qf\"" Nov 03 19:39:31.154720 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.154701 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fx66q\"" Nov 03 19:39:31.154784 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.154722 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 19:39:31.154784 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.154749 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 19:39:31.154885 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.154802 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q82hb\"" Nov 03 19:39:31.155948 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.155930 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6gncv\"" Nov 03 19:39:31.156011 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.155977 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-p5v54\"" Nov 03 19:39:31.156212 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.156197 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:31.156548 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.156518 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-st4g7\"" Nov 03 19:39:31.156691 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.156674 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-fk8jl\"" Nov 03 19:39:31.158249 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.158215 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.158249 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.158225 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.158405 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.158227 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.158405 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.158287 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 19:39:31.158405 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.158335 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 19:39:31.158405 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.158344 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5rk8\"" Nov 03 19:39:31.158405 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.158356 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.158405 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.158394 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 19:39:31.159619 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.159599 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zzp2g\"" Nov 03 19:39:31.159670 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.159648 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 19:39:31.159701 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.159673 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 19:39:31.159862 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.159850 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 19:39:31.159924 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.159889 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 19:39:31.160393 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.160373 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 19:39:31.160393 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.160389 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rq9xl\"" Nov 03 19:39:31.160546 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.160415 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 19:39:31.161927 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.161906 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 19:39:31.162039 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.162024 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 19:39:31.162137 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.162117 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f2rb8\"" Nov 03 19:39:31.162213 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.162197 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zcfm6\"" Nov 03 19:39:31.162299 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.162275 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xbmvx\"" Nov 03 19:39:31.162299 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.162296 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 19:39:31.162412 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.162381 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 19:39:31.162473 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.162446 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 19:39:31.162473 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.162451 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 19:39:31.162614 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.162549 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.162614 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.162594 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 19:39:31.162711 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.162661 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d4wgq\"" Nov 03 19:39:31.162761 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.162754 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.162869 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.162853 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7xvgk\"" Nov 03 19:39:31.162930 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.162919 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.163074 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.163060 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rlc2f\"" Nov 03 19:39:31.163175 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.163163 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 19:39:31.163402 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.163384 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.163485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.163386 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 19:39:31.163485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.163394 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.163485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.163457 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.164025 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164003 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zxqwg\"" Nov 03 19:39:31.164130 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164052 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.164130 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164094 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 19:39:31.164229 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164158 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 19:39:31.164229 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164185 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 19:39:31.164229 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164216 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 19:39:31.164380 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164225 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-56c2j\"" Nov 03 19:39:31.164380 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164284 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vmwsk\"" Nov 03 19:39:31.164380 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164344 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.164380 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164369 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 19:39:31.164577 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164242 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.164577 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164497 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.164577 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164517 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.164577 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164158 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 19:39:31.164706 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164518 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mvxmh\"" Nov 03 19:39:31.164706 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164648 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 19:39:31.164706 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164663 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 19:39:31.164706 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164685 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tlzjx\"" Nov 03 19:39:31.164846 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164751 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jhn8f\"" Nov 03 19:39:31.164846 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164814 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7hnqg\"" Nov 03 19:39:31.164846 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164843 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 19:39:31.164952 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164855 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-drbgz\"" Nov 03 19:39:31.164985 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.164954 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 19:39:31.165013 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.165004 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.165082 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.165070 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 19:39:31.198112 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.198087 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pvs99_e90fc007-32bc-43c5-99c6-7dbb3f9753d7/node-ca/0.log" Nov 03 19:39:31.236858 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.236829 5111 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 19:39:31.248225 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248201 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4734a1de-ee4e-42ab-b925-1a85f65b7d24-cert\") pod \"ingress-canary-ggbpk\" (UID: \"4734a1de-ee4e-42ab-b925-1a85f65b7d24\") " pod="openshift-ingress-canary/ingress-canary-ggbpk" Nov 03 19:39:31.248370 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248233 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c67h2\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-kube-api-access-c67h2\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.248370 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248254 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-kubelet\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.248370 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248277 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysctl-d\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.248370 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248317 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4bgf\" (UniqueName: \"kubernetes.io/projected/0813f12f-0d14-47a8-b4b0-76b159b4e845-kube-api-access-v4bgf\") pod \"migrator-66df5d496b-t8rfb\" (UID: \"0813f12f-0d14-47a8-b4b0-76b159b4e845\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t8rfb" Nov 03 19:39:31.248604 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248476 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cni-binary-copy\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.248604 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248499 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-stats-auth\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:31.248604 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248547 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:31.248604 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248565 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4734a1de-ee4e-42ab-b925-1a85f65b7d24-cert\") pod \"ingress-canary-ggbpk\" (UID: \"4734a1de-ee4e-42ab-b925-1a85f65b7d24\") " pod="openshift-ingress-canary/ingress-canary-ggbpk" Nov 03 19:39:31.248799 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248607 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6bab2991-93a7-44c8-9071-e3c971b79885-ca-trust-extracted\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.248799 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248632 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-device-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.248799 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248677 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr8k2\" (UniqueName: \"kubernetes.io/projected/04f022dd-949d-40ab-b00c-902e70ef610e-kube-api-access-gr8k2\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.248799 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248725 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-lib-modules\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.248799 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248750 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-slash\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.248799 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248753 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6bab2991-93a7-44c8-9071-e3c971b79885-ca-trust-extracted\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.248799 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248775 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-var-lib-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.248799 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248795 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cni-binary-copy\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.249132 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248802 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-stats-auth\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:31.249132 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248808 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.249132 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248843 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5101a457-b490-4538-a9f6-288014fa9264-config-volume\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:31.249132 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248873 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-tuned\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.249132 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248895 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hcnk\" (UniqueName: \"kubernetes.io/projected/2266dada-9604-44f9-b488-511006cee7b2-kube-api-access-2hcnk\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:31.249132 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248944 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/edae5b46-8158-4dc8-91c9-06ed65126db9-dbus\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:31.249132 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.248979 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-tuned\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.249132 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249112 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5101a457-b490-4538-a9f6-288014fa9264-config-volume\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:31.249481 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249133 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-sys-fs\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.249481 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249179 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sk4zs\" (UID: \"b2503201-686a-4cf9-b5a1-977c87e6b717\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:31.249481 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249207 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-bound-sa-token\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.249481 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249227 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ca34d416-e238-48cb-a507-953f78fead22-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:31.249481 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249257 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cnibin\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.249481 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249279 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-signing-key\") pod \"service-ca-5fffc97bdf-fckg5\" (UID: \"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:31.249481 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249301 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hnsk\" (UniqueName: \"kubernetes.io/projected/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-api-access-8hnsk\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:31.249481 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249320 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25n72\" (UniqueName: \"kubernetes.io/projected/51c48ea1-6b42-4ec0-ae95-bf81b913e901-kube-api-access-25n72\") pod \"kas-connection-checker-685895c7c9-d5qxk\" (UID: \"51c48ea1-6b42-4ec0-ae95-bf81b913e901\") " pod="kube-system/kas-connection-checker-685895c7c9-d5qxk" Nov 03 19:39:31.249481 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249337 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs\") pod \"network-metrics-daemon-z9svh\" (UID: \"2dcde652-ceea-4a58-9746-eb2087dea702\") " pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:31.249481 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249361 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/398cf284-004e-463a-a74e-0193946df8b7-cni-binary-copy\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.249481 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249427 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-netns\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.250010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249491 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ca34d416-e238-48cb-a507-953f78fead22-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:31.250010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249505 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-archives-path\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:31.250010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249575 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73f2218a-88f7-4f26-98b3-2d6403bff02d-config\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:31.250010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249630 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-archives-path\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:31.250010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249649 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-signing-key\") pod \"service-ca-5fffc97bdf-fckg5\" (UID: \"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:31.250010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249657 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.250010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249714 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zcvd\" (UniqueName: \"kubernetes.io/projected/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-kube-api-access-7zcvd\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:31.250010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249738 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/398cf284-004e-463a-a74e-0193946df8b7-cni-binary-copy\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.250010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249750 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgc9h\" (UniqueName: \"kubernetes.io/projected/5101a457-b490-4538-a9f6-288014fa9264-kube-api-access-rgc9h\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:31.250010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249763 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73f2218a-88f7-4f26-98b3-2d6403bff02d-config\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:31.250010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249773 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-systemd\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.250010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249800 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:31.250010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249838 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57r2f\" (UniqueName: \"kubernetes.io/projected/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-kube-api-access-57r2f\") pod \"node-ca-pvs99\" (UID: \"e90fc007-32bc-43c5-99c6-7dbb3f9753d7\") " pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:39:31.250010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249868 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwnxx\" (UniqueName: \"kubernetes.io/projected/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-kube-api-access-hwnxx\") pod \"service-ca-5fffc97bdf-fckg5\" (UID: \"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:31.250010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.249971 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/754b8c74-0816-45c8-92f6-95725095e262-ovn-node-metrics-cert\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.250010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250004 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73f2218a-88f7-4f26-98b3-2d6403bff02d-trusted-ca\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250041 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250046 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250067 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-ovn\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250091 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250140 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-var-lib-kubelet\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250211 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgn5q\" (UniqueName: \"kubernetes.io/projected/44cfa442-7cca-4e81-af9c-789fb885a12c-kube-api-access-dgn5q\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250286 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-installation-pull-secrets\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250316 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250377 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/754b8c74-0816-45c8-92f6-95725095e262-ovn-node-metrics-cert\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250386 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-host\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250416 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22vq\" (UniqueName: \"kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq\") pod \"network-check-target-wztqw\" (UID: \"d85c1bad-d10e-4276-9471-0fbe78bb67b3\") " pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250448 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73f2218a-88f7-4f26-98b3-2d6403bff02d-trusted-ca\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250442 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-cni-netd\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250566 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74a9a94d-e73f-4942-bab6-efbfcb728df2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p\" (UID: \"74a9a94d-e73f-4942-bab6-efbfcb728df2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250588 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-registry-certificates\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250606 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73f2218a-88f7-4f26-98b3-2d6403bff02d-serving-cert\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250624 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pp9v\" (UniqueName: \"kubernetes.io/projected/73f2218a-88f7-4f26-98b3-2d6403bff02d-kube-api-access-9pp9v\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:31.250664 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250664 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-installation-pull-secrets\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.251134 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250678 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-trusted-ca\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.251134 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250821 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4hfr\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-kube-api-access-g4hfr\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.251134 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250840 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-systemd-units\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.251134 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250860 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jz64\" (UniqueName: \"kubernetes.io/projected/6efc8dc6-6157-4518-b5fc-dfa8af652691-kube-api-access-2jz64\") pod \"migrator-66df5d496b-slmpf\" (UID: \"6efc8dc6-6157-4518-b5fc-dfa8af652691\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-slmpf" Nov 03 19:39:31.251134 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250893 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-signing-cabundle\") pod \"service-ca-5fffc97bdf-fckg5\" (UID: \"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:31.251134 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250929 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg6c5\" (UniqueName: \"kubernetes.io/projected/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-kube-api-access-dg6c5\") pod \"iptables-alerter-rhlgs\" (UID: \"d9e5c9ec-1940-4cb2-8c55-12da74cfc620\") " pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:39:31.251134 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250943 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-trusted-ca\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.251134 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250946 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-service-ca-bundle\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:31.251134 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.250977 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-ovnkube-config\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.251134 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251025 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19cae502-56f5-4ec2-9867-205025a68d83-agent-certs\") pod \"konnectivity-agent-f4x7r\" (UID: \"19cae502-56f5-4ec2-9867-205025a68d83\") " pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:39:31.251134 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251042 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-multus-certs\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.251134 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251063 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl29m\" (UniqueName: \"kubernetes.io/projected/398cf284-004e-463a-a74e-0193946df8b7-kube-api-access-zl29m\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.251444 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251150 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-service-ca-bundle\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:31.251444 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251156 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d134b335-9e0e-41c9-a676-566a6ef55f5a-data-volume\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:31.251444 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251179 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-image-registry-private-configuration\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.251444 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251196 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/398cf284-004e-463a-a74e-0193946df8b7-multus-daemon-config\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.251444 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251213 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-certificates\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.251444 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251228 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gk22\" (UniqueName: \"kubernetes.io/projected/9924fd9e-f7b6-4036-8584-22caa1de39b3-kube-api-access-7gk22\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:31.251444 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251245 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-trusted-ca-bundle\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:31.251444 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251260 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qx5n\" (UniqueName: \"kubernetes.io/projected/ca34d416-e238-48cb-a507-953f78fead22-kube-api-access-4qx5n\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:31.251444 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251282 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fvg7\" (UniqueName: \"kubernetes.io/projected/257528a3-a0f4-4a07-bba0-67561f8ee205-kube-api-access-9fvg7\") pod \"kas-connection-checker-685895c7c9-x9cw5\" (UID: \"257528a3-a0f4-4a07-bba0-67561f8ee205\") " pod="kube-system/kas-connection-checker-685895c7c9-x9cw5" Nov 03 19:39:31.251444 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251312 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-cni-multus\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.251444 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251402 5111 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:31.251444 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251434 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74a9a94d-e73f-4942-bab6-efbfcb728df2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p\" (UID: \"74a9a94d-e73f-4942-bab6-efbfcb728df2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:31.251947 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251615 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-signing-cabundle\") pod \"service-ca-5fffc97bdf-fckg5\" (UID: \"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:31.251947 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251654 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysctl-conf\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.251947 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251665 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d134b335-9e0e-41c9-a676-566a6ef55f5a-data-volume\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:31.251947 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251688 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2266dada-9604-44f9-b488-511006cee7b2-serving-cert\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:31.251947 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251713 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:31.251947 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251738 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-etc-selinux\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.251947 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251754 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-ovnkube-config\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.251947 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251767 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-run-netns\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.251947 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251663 5111 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:31.251947 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251840 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-image-registry-private-configuration\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.251947 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251895 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73f2218a-88f7-4f26-98b3-2d6403bff02d-serving-cert\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:31.252401 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.251969 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-trusted-ca-bundle\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:31.252401 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252012 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-node-log\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.252401 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252022 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2266dada-9604-44f9-b488-511006cee7b2-serving-cert\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:31.252401 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252075 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-run-ovn-kubernetes\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.252401 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252084 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/398cf284-004e-463a-a74e-0193946df8b7-multus-daemon-config\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.252401 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252127 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-hosts-file\") pod \"node-resolver-k666m\" (UID: \"29f79e6b-20d5-4ba4-8737-068f0d82b1e6\") " pod="openshift-dns/node-resolver-k666m" Nov 03 19:39:31.252401 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252163 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-cnibin\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.252401 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252198 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-conf-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.252401 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252234 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-host\") pod \"node-ca-pvs99\" (UID: \"e90fc007-32bc-43c5-99c6-7dbb3f9753d7\") " pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:39:31.252401 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252281 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19cae502-56f5-4ec2-9867-205025a68d83-agent-certs\") pod \"konnectivity-agent-f4x7r\" (UID: \"19cae502-56f5-4ec2-9867-205025a68d83\") " pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:39:31.252401 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252323 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-ca-trust-extracted\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.252755 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252417 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-ca-trust-extracted\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.252755 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252448 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.252755 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252467 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-sys\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.252755 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252490 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2266dada-9604-44f9-b488-511006cee7b2-tmp\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:31.252755 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252578 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-certificates\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.252755 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252609 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2266dada-9604-44f9-b488-511006cee7b2-tmp\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:31.252755 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252609 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2266dada-9604-44f9-b488-511006cee7b2-snapshots\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:31.252755 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252656 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-image-registry-private-configuration\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.252755 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252676 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.252755 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252692 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69c72eb2-3839-4483-82e8-b7d0bf82c57e-serving-cert\") pod \"service-ca-operator-544d749c8c-bqmb8\" (UID: \"69c72eb2-3839-4483-82e8-b7d0bf82c57e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:31.252755 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252676 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-registry-certificates\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.253112 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252791 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-iptables-alerter-script\") pod \"iptables-alerter-rhlgs\" (UID: \"d9e5c9ec-1940-4cb2-8c55-12da74cfc620\") " pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:39:31.253112 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252801 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2266dada-9604-44f9-b488-511006cee7b2-snapshots\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:31.253112 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252818 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-kubelet\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.253112 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252835 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-image-registry-private-configuration\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.253112 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252852 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-k8s-cni-cncf-io\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.253112 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252895 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.253112 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252918 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-cni-bin\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.253112 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252971 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-log-socket\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.253112 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.252996 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-ovnkube-script-lib\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.253112 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253021 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-tmp-dir\") pod \"node-resolver-k666m\" (UID: \"29f79e6b-20d5-4ba4-8737-068f0d82b1e6\") " pod="openshift-dns/node-resolver-k666m" Nov 03 19:39:31.253112 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253048 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5101a457-b490-4538-a9f6-288014fa9264-tmp-dir\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:31.253112 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253076 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-installation-pull-secrets\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.253112 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253093 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69c72eb2-3839-4483-82e8-b7d0bf82c57e-serving-cert\") pod \"service-ca-operator-544d749c8c-bqmb8\" (UID: \"69c72eb2-3839-4483-82e8-b7d0bf82c57e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:31.253112 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253092 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-iptables-alerter-script\") pod \"iptables-alerter-rhlgs\" (UID: \"d9e5c9ec-1940-4cb2-8c55-12da74cfc620\") " pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:39:31.253112 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253104 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253119 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-tmp-dir\") pod \"node-resolver-k666m\" (UID: \"29f79e6b-20d5-4ba4-8737-068f0d82b1e6\") " pod="openshift-dns/node-resolver-k666m" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253130 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253155 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5101a457-b490-4538-a9f6-288014fa9264-tmp-dir\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253173 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253197 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-ovnkube-script-lib\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253199 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-systemd\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253269 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-installation-pull-secrets\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253266 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-socket-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253312 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysconfig\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253344 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/edae5b46-8158-4dc8-91c9-06ed65126db9-kubelet-config\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253369 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-env-overrides\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253395 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edae5b46-8158-4dc8-91c9-06ed65126db9-original-pull-secret\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253472 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dph2k\" (UniqueName: \"kubernetes.io/projected/d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041-kube-api-access-dph2k\") pod \"kas-connection-checker-685895c7c9-knv7j\" (UID: \"d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041\") " pod="kube-system/kas-connection-checker-685895c7c9-knv7j" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253477 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/754b8c74-0816-45c8-92f6-95725095e262-env-overrides\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253494 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7rsm\" (UniqueName: \"kubernetes.io/projected/754b8c74-0816-45c8-92f6-95725095e262-kube-api-access-b7rsm\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253512 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdcb\" (UniqueName: \"kubernetes.io/projected/4734a1de-ee4e-42ab-b925-1a85f65b7d24-kube-api-access-9hdcb\") pod \"ingress-canary-ggbpk\" (UID: \"4734a1de-ee4e-42ab-b925-1a85f65b7d24\") " pod="openshift-ingress-canary/ingress-canary-ggbpk" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253553 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253666 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-system-cni-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253701 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2503201-686a-4cf9-b5a1-977c87e6b717-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sk4zs\" (UID: \"b2503201-686a-4cf9-b5a1-977c87e6b717\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:31.253739 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253722 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7qbv\" (UniqueName: \"kubernetes.io/projected/09d5c2e9-13cd-4859-abbf-218d18877c5a-kube-api-access-s7qbv\") pod \"network-check-source-6f78d59f99-f9vq4\" (UID: \"09d5c2e9-13cd-4859-abbf-218d18877c5a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f9vq4" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253762 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-host-slash\") pod \"iptables-alerter-rhlgs\" (UID: \"d9e5c9ec-1940-4cb2-8c55-12da74cfc620\") " pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253781 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253798 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74a9a94d-e73f-4942-bab6-efbfcb728df2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p\" (UID: \"74a9a94d-e73f-4942-bab6-efbfcb728df2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253812 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-hostroot\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253850 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-etc-kubernetes\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253888 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-serviceca\") pod \"node-ca-pvs99\" (UID: \"e90fc007-32bc-43c5-99c6-7dbb3f9753d7\") " pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253904 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2503201-686a-4cf9-b5a1-977c87e6b717-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sk4zs\" (UID: \"b2503201-686a-4cf9-b5a1-977c87e6b717\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.253910 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-system-cni-dir\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254033 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-567td\" (UniqueName: \"kubernetes.io/projected/36460aa5-b1f0-4be4-b27b-675c6fecffe1-kube-api-access-567td\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254034 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74a9a94d-e73f-4942-bab6-efbfcb728df2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p\" (UID: \"74a9a94d-e73f-4942-bab6-efbfcb728df2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254084 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-etc-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254114 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5j6b\" (UniqueName: \"kubernetes.io/projected/0fbfc914-8c97-4770-9ef6-6e553c4dcc58-kube-api-access-r5j6b\") pod \"volume-data-source-validator-58c9f5f46d-txtq2\" (UID: \"0fbfc914-8c97-4770-9ef6-6e553c4dcc58\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-txtq2" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254144 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-bound-sa-token\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254168 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44cfa442-7cca-4e81-af9c-789fb885a12c-tmp\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254211 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz9vj\" (UniqueName: \"kubernetes.io/projected/2dcde652-ceea-4a58-9746-eb2087dea702-kube-api-access-jz9vj\") pod \"network-metrics-daemon-z9svh\" (UID: \"2dcde652-ceea-4a58-9746-eb2087dea702\") " pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254270 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-trusted-ca\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254289 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44cfa442-7cca-4e81-af9c-789fb885a12c-tmp\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254297 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-registration-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254339 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254376 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254412 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19cae502-56f5-4ec2-9867-205025a68d83-konnectivity-ca\") pod \"konnectivity-agent-f4x7r\" (UID: \"19cae502-56f5-4ec2-9867-205025a68d83\") " pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:39:31.254488 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254438 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2266dada-9604-44f9-b488-511006cee7b2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254509 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-serviceca\") pod \"node-ca-pvs99\" (UID: \"e90fc007-32bc-43c5-99c6-7dbb3f9753d7\") " pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254576 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2266dada-9604-44f9-b488-511006cee7b2-service-ca-bundle\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254604 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-cni-bin\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254629 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-cni-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254629 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca34d416-e238-48cb-a507-953f78fead22-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254639 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/36460aa5-b1f0-4be4-b27b-675c6fecffe1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254655 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-socket-dir-parent\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254659 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19cae502-56f5-4ec2-9867-205025a68d83-konnectivity-ca\") pod \"konnectivity-agent-f4x7r\" (UID: \"19cae502-56f5-4ec2-9867-205025a68d83\") " pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254693 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-trusted-ca\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254694 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-modprobe-d\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254735 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-kubernetes\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254756 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-run\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254774 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4w67\" (UniqueName: \"kubernetes.io/projected/69c72eb2-3839-4483-82e8-b7d0bf82c57e-kube-api-access-f4w67\") pod \"service-ca-operator-544d749c8c-bqmb8\" (UID: \"69c72eb2-3839-4483-82e8-b7d0bf82c57e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254800 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d134b335-9e0e-41c9-a676-566a6ef55f5a-crio-socket\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254806 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2266dada-9604-44f9-b488-511006cee7b2-service-ca-bundle\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254819 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8x2d\" (UniqueName: \"kubernetes.io/projected/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-kube-api-access-f8x2d\") pod \"node-resolver-k666m\" (UID: \"29f79e6b-20d5-4ba4-8737-068f0d82b1e6\") " pod="openshift-dns/node-resolver-k666m" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254797 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2266dada-9604-44f9-b488-511006cee7b2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254919 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvzw\" (UniqueName: \"kubernetes.io/projected/74a9a94d-e73f-4942-bab6-efbfcb728df2-kube-api-access-mmvzw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p\" (UID: \"74a9a94d-e73f-4942-bab6-efbfcb728df2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254936 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpk6g\" (UniqueName: \"kubernetes.io/projected/1b0776e1-a654-4e61-ba94-6d81ddfc8877-kube-api-access-fpk6g\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.254975 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/69c72eb2-3839-4483-82e8-b7d0bf82c57e-config\") pod \"service-ca-operator-544d749c8c-bqmb8\" (UID: \"69c72eb2-3839-4483-82e8-b7d0bf82c57e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:31.255172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.255072 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-default-certificate\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:31.255780 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.255269 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/69c72eb2-3839-4483-82e8-b7d0bf82c57e-config\") pod \"service-ca-operator-544d749c8c-bqmb8\" (UID: \"69c72eb2-3839-4483-82e8-b7d0bf82c57e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:31.255780 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.255377 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-default-certificate\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:31.257714 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.257304 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hnsk\" (UniqueName: \"kubernetes.io/projected/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-api-access-8hnsk\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:31.257714 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.257328 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25n72\" (UniqueName: \"kubernetes.io/projected/51c48ea1-6b42-4ec0-ae95-bf81b913e901-kube-api-access-25n72\") pod \"kas-connection-checker-685895c7c9-d5qxk\" (UID: \"51c48ea1-6b42-4ec0-ae95-bf81b913e901\") " pod="kube-system/kas-connection-checker-685895c7c9-d5qxk" Nov 03 19:39:31.257714 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.257405 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr8k2\" (UniqueName: \"kubernetes.io/projected/04f022dd-949d-40ab-b00c-902e70ef610e-kube-api-access-gr8k2\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.257714 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.257449 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hcnk\" (UniqueName: \"kubernetes.io/projected/2266dada-9604-44f9-b488-511006cee7b2-kube-api-access-2hcnk\") pod \"insights-operator-7c4986d449-rs8wk\" (UID: \"2266dada-9604-44f9-b488-511006cee7b2\") " pod="openshift-insights/insights-operator-7c4986d449-rs8wk" Nov 03 19:39:31.257714 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.257674 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-bound-sa-token\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.257714 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.257705 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c67h2\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-kube-api-access-c67h2\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.258018 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.257765 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zcvd\" (UniqueName: \"kubernetes.io/projected/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-kube-api-access-7zcvd\") pod \"periodic-gathering-fn7j5-kgmpq\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:39:31.258018 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.257913 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwnxx\" (UniqueName: \"kubernetes.io/projected/6fd367f3-a564-4922-8d7b-2d4b84eaa1bc-kube-api-access-hwnxx\") pod \"service-ca-5fffc97bdf-fckg5\" (UID: \"6fd367f3-a564-4922-8d7b-2d4b84eaa1bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fckg5" Nov 03 19:39:31.258726 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.258709 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57r2f\" (UniqueName: \"kubernetes.io/projected/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-kube-api-access-57r2f\") pod \"node-ca-pvs99\" (UID: \"e90fc007-32bc-43c5-99c6-7dbb3f9753d7\") " pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:39:31.259930 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.259911 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4bgf\" (UniqueName: \"kubernetes.io/projected/0813f12f-0d14-47a8-b4b0-76b159b4e845-kube-api-access-v4bgf\") pod \"migrator-66df5d496b-t8rfb\" (UID: \"0813f12f-0d14-47a8-b4b0-76b159b4e845\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t8rfb" Nov 03 19:39:31.260674 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.260649 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl29m\" (UniqueName: \"kubernetes.io/projected/398cf284-004e-463a-a74e-0193946df8b7-kube-api-access-zl29m\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.260935 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.260911 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pp9v\" (UniqueName: \"kubernetes.io/projected/73f2218a-88f7-4f26-98b3-2d6403bff02d-kube-api-access-9pp9v\") pod \"console-operator-7cd56b8dbc-68j2z\" (UID: \"73f2218a-88f7-4f26-98b3-2d6403bff02d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:31.261045 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.260941 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgc9h\" (UniqueName: \"kubernetes.io/projected/5101a457-b490-4538-a9f6-288014fa9264-kube-api-access-rgc9h\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:31.261045 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.260976 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jz64\" (UniqueName: \"kubernetes.io/projected/6efc8dc6-6157-4518-b5fc-dfa8af652691-kube-api-access-2jz64\") pod \"migrator-66df5d496b-slmpf\" (UID: \"6efc8dc6-6157-4518-b5fc-dfa8af652691\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-slmpf" Nov 03 19:39:31.261118 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.261056 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qx5n\" (UniqueName: \"kubernetes.io/projected/ca34d416-e238-48cb-a507-953f78fead22-kube-api-access-4qx5n\") pod \"cluster-monitoring-operator-884576c57-zx2m4\" (UID: \"ca34d416-e238-48cb-a507-953f78fead22\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zx2m4" Nov 03 19:39:31.261243 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.261197 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gk22\" (UniqueName: \"kubernetes.io/projected/9924fd9e-f7b6-4036-8584-22caa1de39b3-kube-api-access-7gk22\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:31.261366 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.261343 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgn5q\" (UniqueName: \"kubernetes.io/projected/44cfa442-7cca-4e81-af9c-789fb885a12c-kube-api-access-dgn5q\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.261429 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.261390 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg6c5\" (UniqueName: \"kubernetes.io/projected/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-kube-api-access-dg6c5\") pod \"iptables-alerter-rhlgs\" (UID: \"d9e5c9ec-1940-4cb2-8c55-12da74cfc620\") " pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:39:31.261606 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.261588 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4hfr\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-kube-api-access-g4hfr\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.261758 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.261740 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fvg7\" (UniqueName: \"kubernetes.io/projected/257528a3-a0f4-4a07-bba0-67561f8ee205-kube-api-access-9fvg7\") pod \"kas-connection-checker-685895c7c9-x9cw5\" (UID: \"257528a3-a0f4-4a07-bba0-67561f8ee205\") " pod="kube-system/kas-connection-checker-685895c7c9-x9cw5" Nov 03 19:39:31.261841 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.261821 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k22vq\" (UniqueName: \"kubernetes.io/projected/d85c1bad-d10e-4276-9471-0fbe78bb67b3-kube-api-access-k22vq\") pod \"network-check-target-wztqw\" (UID: \"d85c1bad-d10e-4276-9471-0fbe78bb67b3\") " pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:31.265770 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.265755 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz9vj\" (UniqueName: \"kubernetes.io/projected/2dcde652-ceea-4a58-9746-eb2087dea702-kube-api-access-jz9vj\") pod \"network-metrics-daemon-z9svh\" (UID: \"2dcde652-ceea-4a58-9746-eb2087dea702\") " pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:31.268424 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.268404 5111 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:39:31.268506 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.268440 5111 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:31.268913 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.268897 5111 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:39:31.268987 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.268943 5111 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-178.us-east-2.compute.internal" Nov 03 19:39:31.269996 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.269978 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-567td\" (UniqueName: \"kubernetes.io/projected/36460aa5-b1f0-4be4-b27b-675c6fecffe1-kube-api-access-567td\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.270927 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.270694 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7rsm\" (UniqueName: \"kubernetes.io/projected/754b8c74-0816-45c8-92f6-95725095e262-kube-api-access-b7rsm\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.270927 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.270872 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpk6g\" (UniqueName: \"kubernetes.io/projected/1b0776e1-a654-4e61-ba94-6d81ddfc8877-kube-api-access-fpk6g\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:31.270927 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.270908 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7qbv\" (UniqueName: \"kubernetes.io/projected/09d5c2e9-13cd-4859-abbf-218d18877c5a-kube-api-access-s7qbv\") pod \"network-check-source-6f78d59f99-f9vq4\" (UID: \"09d5c2e9-13cd-4859-abbf-218d18877c5a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f9vq4" Nov 03 19:39:31.270927 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.270923 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hdcb\" (UniqueName: \"kubernetes.io/projected/4734a1de-ee4e-42ab-b925-1a85f65b7d24-kube-api-access-9hdcb\") pod \"ingress-canary-ggbpk\" (UID: \"4734a1de-ee4e-42ab-b925-1a85f65b7d24\") " pod="openshift-ingress-canary/ingress-canary-ggbpk" Nov 03 19:39:31.271122 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.270906 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvzw\" (UniqueName: \"kubernetes.io/projected/74a9a94d-e73f-4942-bab6-efbfcb728df2-kube-api-access-mmvzw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p\" (UID: \"74a9a94d-e73f-4942-bab6-efbfcb728df2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" Nov 03 19:39:31.271208 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.271184 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5j6b\" (UniqueName: \"kubernetes.io/projected/0fbfc914-8c97-4770-9ef6-6e553c4dcc58-kube-api-access-r5j6b\") pod \"volume-data-source-validator-58c9f5f46d-txtq2\" (UID: \"0fbfc914-8c97-4770-9ef6-6e553c4dcc58\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-txtq2" Nov 03 19:39:31.271287 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.271263 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8x2d\" (UniqueName: \"kubernetes.io/projected/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-kube-api-access-f8x2d\") pod \"node-resolver-k666m\" (UID: \"29f79e6b-20d5-4ba4-8737-068f0d82b1e6\") " pod="openshift-dns/node-resolver-k666m" Nov 03 19:39:31.271320 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.271288 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-bound-sa-token\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.271385 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.271324 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4w67\" (UniqueName: \"kubernetes.io/projected/69c72eb2-3839-4483-82e8-b7d0bf82c57e-kube-api-access-f4w67\") pod \"service-ca-operator-544d749c8c-bqmb8\" (UID: \"69c72eb2-3839-4483-82e8-b7d0bf82c57e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" Nov 03 19:39:31.271385 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.271376 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dph2k\" (UniqueName: \"kubernetes.io/projected/d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041-kube-api-access-dph2k\") pod \"kas-connection-checker-685895c7c9-knv7j\" (UID: \"d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041\") " pod="kube-system/kas-connection-checker-685895c7c9-knv7j" Nov 03 19:39:31.355981 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.355943 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.355981 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.355980 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-system-cni-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.356187 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.355997 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-host-slash\") pod \"iptables-alerter-rhlgs\" (UID: \"d9e5c9ec-1940-4cb2-8c55-12da74cfc620\") " pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:39:31.356187 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356014 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.356187 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356058 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.356187 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356084 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9e5c9ec-1940-4cb2-8c55-12da74cfc620-host-slash\") pod \"iptables-alerter-rhlgs\" (UID: \"d9e5c9ec-1940-4cb2-8c55-12da74cfc620\") " pod="openshift-network-operator/iptables-alerter-rhlgs" Nov 03 19:39:31.356187 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.356099 5111 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:39:31.356187 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356087 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-system-cni-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.356187 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.356120 5111 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b48f6466f-rvj5f: secret "image-registry-tls" not found Nov 03 19:39:31.356187 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356124 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-hostroot\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.356187 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356098 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-hostroot\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.356187 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.356190 5111 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls podName:6bab2991-93a7-44c8-9071-e3c971b79885 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:31.856168952 +0000 UTC m=+1.818547393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls") pod "image-registry-b48f6466f-rvj5f" (UID: "6bab2991-93a7-44c8-9071-e3c971b79885") : secret "image-registry-tls" not found Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356212 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-etc-kubernetes\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356233 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-system-cni-dir\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356249 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-etc-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356269 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-etc-kubernetes\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356289 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-etc-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356297 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-system-cni-dir\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356333 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-registration-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356364 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-cni-bin\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356388 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-cni-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356406 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-registration-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356410 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-socket-dir-parent\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356423 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-cni-bin\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356434 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-modprobe-d\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356455 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-cni-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356464 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-kubernetes\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356482 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-run\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356509 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d134b335-9e0e-41c9-a676-566a6ef55f5a-crio-socket\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356523 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-kubernetes\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356482 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-socket-dir-parent\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.356527 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356561 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-kubelet\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356567 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-run\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356573 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-modprobe-d\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356584 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysctl-d\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356602 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356610 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-kubelet\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356611 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d134b335-9e0e-41c9-a676-566a6ef55f5a-crio-socket\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356619 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-device-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356654 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-device-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356668 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-lib-modules\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356686 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysctl-d\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356694 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-slash\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356717 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-slash\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356719 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-var-lib-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356747 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356772 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356776 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/edae5b46-8158-4dc8-91c9-06ed65126db9-dbus\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.356697 5111 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356787 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-lib-modules\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356750 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-var-lib-openvswitch\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356801 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-sys-fs\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356830 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sk4zs\" (UID: \"b2503201-686a-4cf9-b5a1-977c87e6b717\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356843 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-sys-fs\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356850 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/edae5b46-8158-4dc8-91c9-06ed65126db9-dbus\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.356847 5111 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls podName:d134b335-9e0e-41c9-a676-566a6ef55f5a nodeName:}" failed. No retries permitted until 2026-11-03 19:39:31.856831932 +0000 UTC m=+1.819210368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-md5d2" (UID: "d134b335-9e0e-41c9-a676-566a6ef55f5a") : secret "insights-runtime-extractor-tls" not found Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356888 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cnibin\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356904 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs\") pod \"network-metrics-daemon-z9svh\" (UID: \"2dcde652-ceea-4a58-9746-eb2087dea702\") " pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356928 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-netns\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356946 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-cnibin\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356947 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-netns\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356962 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.356985 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-systemd\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357013 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357037 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-ovn\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357044 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-systemd\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357059 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357084 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-var-lib-kubelet\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357109 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.357088 5111 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357129 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357132 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-host\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357170 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-cni-netd\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.357143 5111 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85444c64c6-qk7p8: secret "image-registry-tls" not found Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.357179 5111 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357182 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-var-lib-kubelet\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357188 5111 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357201 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-systemd-units\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.357214 5111 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls podName:ac78f686-0d5a-4ff1-8a04-101e5794b7d7 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:31.857201595 +0000 UTC m=+1.819580035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls") pod "image-registry-85444c64c6-qk7p8" (UID: "ac78f686-0d5a-4ff1-8a04-101e5794b7d7") : secret "image-registry-tls" not found Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357172 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-host\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357224 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-cni-netd\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.357232 5111 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm podName:d134b335-9e0e-41c9-a676-566a6ef55f5a nodeName:}" failed. No retries permitted until 2026-11-03 19:39:31.857223825 +0000 UTC m=+1.819602242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-md5d2" (UID: "d134b335-9e0e-41c9-a676-566a6ef55f5a") : configmap "kube-rbac-proxy" not found Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357235 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-systemd-units\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357273 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-ovn\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357274 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-multus-certs\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357306 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-multus-certs\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357311 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-cni-multus\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357344 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-cni-multus\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357379 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysctl-conf\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357417 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357446 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-etc-selinux\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357471 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-run-netns\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.357462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357493 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-node-log\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357518 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-run-ovn-kubernetes\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357562 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-hosts-file\") pod \"node-resolver-k666m\" (UID: \"29f79e6b-20d5-4ba4-8737-068f0d82b1e6\") " pod="openshift-dns/node-resolver-k666m" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357569 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-run-netns\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357573 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-etc-selinux\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357494 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysctl-conf\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357599 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-cnibin\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357635 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-node-log\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357649 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-conf-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357678 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-host\") pod \"node-ca-pvs99\" (UID: \"e90fc007-32bc-43c5-99c6-7dbb3f9753d7\") " pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357695 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-cnibin\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357703 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357708 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-multus-conf-dir\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357651 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29f79e6b-20d5-4ba4-8737-068f0d82b1e6-hosts-file\") pod \"node-resolver-k666m\" (UID: \"29f79e6b-20d5-4ba4-8737-068f0d82b1e6\") " pod="openshift-dns/node-resolver-k666m" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357729 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-sys\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357737 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e90fc007-32bc-43c5-99c6-7dbb3f9753d7-host\") pod \"node-ca-pvs99\" (UID: \"e90fc007-32bc-43c5-99c6-7dbb3f9753d7\") " pod="openshift-image-registry/node-ca-pvs99" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357758 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-kubelet\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357770 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36460aa5-b1f0-4be4-b27b-675c6fecffe1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w6vlx\" (UID: \"36460aa5-b1f0-4be4-b27b-675c6fecffe1\") " pod="openshift-multus/multus-additional-cni-plugins-w6vlx" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357778 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9924fd9e-f7b6-4036-8584-22caa1de39b3-service-ca-bundle\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357787 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-sys\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357680 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-run-ovn-kubernetes\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357782 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-k8s-cni-cncf-io\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357814 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-run-k8s-cni-cncf-io\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357832 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-cni-bin\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357812 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-host-kubelet\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357858 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-log-socket\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357882 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/398cf284-004e-463a-a74e-0193946df8b7-host-var-lib-cni-bin\") pod \"multus-npg2d\" (UID: \"398cf284-004e-463a-a74e-0193946df8b7\") " pod="openshift-multus/multus-npg2d" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357888 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357903 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-log-socket\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357915 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357940 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-systemd\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357968 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-socket-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.357990 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysconfig\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.358014 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/edae5b46-8158-4dc8-91c9-06ed65126db9-kubelet-config\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.358079 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04f022dd-949d-40ab-b00c-902e70ef610e-socket-dir\") pod \"aws-ebs-csi-driver-node-krzjw\" (UID: \"04f022dd-949d-40ab-b00c-902e70ef610e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krzjw" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.358090 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/edae5b46-8158-4dc8-91c9-06ed65126db9-kubelet-config\") pod \"global-pull-secret-syncer-p4t7z\" (UID: \"edae5b46-8158-4dc8-91c9-06ed65126db9\") " pod="kube-system/global-pull-secret-syncer-p4t7z" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.358129 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/754b8c74-0816-45c8-92f6-95725095e262-run-systemd\") pod \"ovnkube-node-tzzd5\" (UID: \"754b8c74-0816-45c8-92f6-95725095e262\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.358136 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44cfa442-7cca-4e81-af9c-789fb885a12c-etc-sysconfig\") pod \"tuned-sks78\" (UID: \"44cfa442-7cca-4e81-af9c-789fb885a12c\") " pod="openshift-cluster-node-tuning-operator/tuned-sks78" Nov 03 19:39:31.360377 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.360353 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5101a457-b490-4538-a9f6-288014fa9264-metrics-tls\") pod \"dns-default-zdf6j\" (UID: \"5101a457-b490-4538-a9f6-288014fa9264\") " pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:31.361396 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.360448 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b0776e1-a654-4e61-ba94-6d81ddfc8877-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-szqck\" (UID: \"1b0776e1-a654-4e61-ba94-6d81ddfc8877\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:31.361396 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.360510 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2503201-686a-4cf9-b5a1-977c87e6b717-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sk4zs\" (UID: \"b2503201-686a-4cf9-b5a1-977c87e6b717\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:31.361396 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.360760 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9924fd9e-f7b6-4036-8584-22caa1de39b3-metrics-certs\") pod \"router-default-855cd4cff-prjqj\" (UID: \"9924fd9e-f7b6-4036-8584-22caa1de39b3\") " pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:31.361579 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.361561 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dcde652-ceea-4a58-9746-eb2087dea702-metrics-certs\") pod \"network-metrics-daemon-z9svh\" (UID: \"2dcde652-ceea-4a58-9746-eb2087dea702\") " pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:31.436623 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.436522 5111 scope.go:122] "RemoveContainer" containerID="cc6f5944bc7c8c6ba63cc9c0e8ce8c89768dca1e2ed1b8fc6ea3065c53dfeb5e" Nov 03 19:39:31.436964 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.436938 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:31.439767 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.439743 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" Nov 03 19:39:31.441757 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.441735 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" Nov 03 19:39:31.442701 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.442683 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z9svh" Nov 03 19:39:31.454468 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.454447 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:31.600276 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.599828 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ggbpk_4734a1de-ee4e-42ab-b925-1a85f65b7d24/serve-healthcheck-canary/0.log" Nov 03 19:39:31.754151 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.754068 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-855cd4cff-prjqj"] Nov 03 19:39:31.758442 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:31.758413 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9924fd9e_f7b6_4036_8584_22caa1de39b3.slice/crio-38222b9f65e108872021e9be44a9f8171f70947ac5363eacb192186d41af3b76 WatchSource:0}: Error finding container 38222b9f65e108872021e9be44a9f8171f70947ac5363eacb192186d41af3b76: Status 404 returned error can't find the container with id 38222b9f65e108872021e9be44a9f8171f70947ac5363eacb192186d41af3b76 Nov 03 19:39:31.800086 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.800066 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-slmpf_6efc8dc6-6157-4518-b5fc-dfa8af652691/migrator/0.log" Nov 03 19:39:31.802125 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.802099 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck"] Nov 03 19:39:31.844824 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.844785 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zdf6j"] Nov 03 19:39:31.847018 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.846984 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-sk4zs"] Nov 03 19:39:31.849429 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:31.849404 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5101a457_b490_4538_a9f6_288014fa9264.slice/crio-903542cb5436c79af2ef1dd11eefd5c28687fe274a03e5bdfa998c0fc633dfa7 WatchSource:0}: Error finding container 903542cb5436c79af2ef1dd11eefd5c28687fe274a03e5bdfa998c0fc633dfa7: Status 404 returned error can't find the container with id 903542cb5436c79af2ef1dd11eefd5c28687fe274a03e5bdfa998c0fc633dfa7 Nov 03 19:39:31.851740 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.850502 5111 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:39:31.852846 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:31.852812 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2503201_686a_4cf9_b5a1_977c87e6b717.slice/crio-5c6bba589bcaf83778e641e021f48e022d8757c652bba89ac09fa5e570d47870 WatchSource:0}: Error finding container 5c6bba589bcaf83778e641e021f48e022d8757c652bba89ac09fa5e570d47870: Status 404 returned error can't find the container with id 5c6bba589bcaf83778e641e021f48e022d8757c652bba89ac09fa5e570d47870 Nov 03 19:39:31.865687 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.865661 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:31.865821 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.865702 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:31.865821 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.865725 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:31.865821 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.865786 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:31.865968 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.865815 5111 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:39:31.865968 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.865846 5111 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:39:31.865968 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.865881 5111 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls podName:d134b335-9e0e-41c9-a676-566a6ef55f5a nodeName:}" failed. No retries permitted until 2026-11-03 19:39:32.865861214 +0000 UTC m=+2.828239630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-md5d2" (UID: "d134b335-9e0e-41c9-a676-566a6ef55f5a") : secret "insights-runtime-extractor-tls" not found Nov 03 19:39:31.865968 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.865887 5111 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:39:31.865968 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.865899 5111 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm podName:d134b335-9e0e-41c9-a676-566a6ef55f5a nodeName:}" failed. No retries permitted until 2026-11-03 19:39:32.865890217 +0000 UTC m=+2.828268637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-md5d2" (UID: "d134b335-9e0e-41c9-a676-566a6ef55f5a") : configmap "kube-rbac-proxy" not found Nov 03 19:39:31.865968 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.865900 5111 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b48f6466f-rvj5f: secret "image-registry-tls" not found Nov 03 19:39:31.865968 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.865922 5111 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:39:31.865968 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.865931 5111 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85444c64c6-qk7p8: secret "image-registry-tls" not found Nov 03 19:39:31.865968 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.865942 5111 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls podName:6bab2991-93a7-44c8-9071-e3c971b79885 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:32.865924719 +0000 UTC m=+2.828303151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls") pod "image-registry-b48f6466f-rvj5f" (UID: "6bab2991-93a7-44c8-9071-e3c971b79885") : secret "image-registry-tls" not found Nov 03 19:39:31.865968 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:31.865972 5111 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls podName:ac78f686-0d5a-4ff1-8a04-101e5794b7d7 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:32.86596439 +0000 UTC m=+2.828342810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls") pod "image-registry-85444c64c6-qk7p8" (UID: "ac78f686-0d5a-4ff1-8a04-101e5794b7d7") : secret "image-registry-tls" not found Nov 03 19:39:31.882104 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.882069 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z9svh"] Nov 03 19:39:31.888184 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:31.888149 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dcde652_ceea_4a58_9746_eb2087dea702.slice/crio-6456b510e175f446a0f675f16f2bff8041b08739073e2e429fe7f566ff617134 WatchSource:0}: Error finding container 6456b510e175f446a0f675f16f2bff8041b08739073e2e429fe7f566ff617134: Status 404 returned error can't find the container with id 6456b510e175f446a0f675f16f2bff8041b08739073e2e429fe7f566ff617134 Nov 03 19:39:31.998587 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:31.998519 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-t8rfb_0813f12f-0d14-47a8-b4b0-76b159b4e845/migrator/0.log" Nov 03 19:39:32.200938 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.200904 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p_74a9a94d-e73f-4942-bab6-efbfcb728df2/kube-storage-version-migrator-operator/0.log" Nov 03 19:39:32.259545 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.259495 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z9svh" event={"ID":"2dcde652-ceea-4a58-9746-eb2087dea702","Type":"ContainerStarted","Data":"6456b510e175f446a0f675f16f2bff8041b08739073e2e429fe7f566ff617134"} Nov 03 19:39:32.260802 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.260772 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zdf6j" event={"ID":"5101a457-b490-4538-a9f6-288014fa9264","Type":"ContainerStarted","Data":"903542cb5436c79af2ef1dd11eefd5c28687fe274a03e5bdfa998c0fc633dfa7"} Nov 03 19:39:32.262255 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.262233 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-855cd4cff-prjqj" event={"ID":"9924fd9e-f7b6-4036-8584-22caa1de39b3","Type":"ContainerStarted","Data":"304c11e6e7ed4d854238a2879874238eefec4aa666fc815800dd458a944fd065"} Nov 03 19:39:32.262342 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.262270 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-855cd4cff-prjqj" event={"ID":"9924fd9e-f7b6-4036-8584-22caa1de39b3","Type":"ContainerStarted","Data":"38222b9f65e108872021e9be44a9f8171f70947ac5363eacb192186d41af3b76"} Nov 03 19:39:32.263673 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.263647 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" event={"ID":"b2503201-686a-4cf9-b5a1-977c87e6b717","Type":"ContainerStarted","Data":"5c6bba589bcaf83778e641e021f48e022d8757c652bba89ac09fa5e570d47870"} Nov 03 19:39:32.264830 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.264807 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" event={"ID":"1b0776e1-a654-4e61-ba94-6d81ddfc8877","Type":"ContainerStarted","Data":"32ec6bec8588660e0d2924acebb660a677c8c423bd2106674ed9d7c5fcaf01e4"} Nov 03 19:39:32.267028 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.267009 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 19:39:32.267107 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.267076 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" event={"ID":"73f2218a-88f7-4f26-98b3-2d6403bff02d","Type":"ContainerStarted","Data":"5278b93c83c5c63cf085e2a174fd1733e9160ae6bf09f7d6af470b5338855a2a"} Nov 03 19:39:32.379919 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.379875 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:39:32.438010 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.437977 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:32.441240 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.441008 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:32.878347 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.878310 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:32.878625 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.878388 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:32.878625 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.878426 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:32.878625 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:32.878458 5111 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:39:32.878625 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:32.878479 5111 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b48f6466f-rvj5f: secret "image-registry-tls" not found Nov 03 19:39:32.878625 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:32.878563 5111 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls podName:6bab2991-93a7-44c8-9071-e3c971b79885 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:34.878513051 +0000 UTC m=+4.840891468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls") pod "image-registry-b48f6466f-rvj5f" (UID: "6bab2991-93a7-44c8-9071-e3c971b79885") : secret "image-registry-tls" not found Nov 03 19:39:32.878901 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.878459 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:32.878987 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:32.878955 5111 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:39:32.879061 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:32.878993 5111 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls podName:d134b335-9e0e-41c9-a676-566a6ef55f5a nodeName:}" failed. No retries permitted until 2026-11-03 19:39:34.878982299 +0000 UTC m=+4.841360717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-md5d2" (UID: "d134b335-9e0e-41c9-a676-566a6ef55f5a") : secret "insights-runtime-extractor-tls" not found Nov 03 19:39:32.879061 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:32.879005 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d134b335-9e0e-41c9-a676-566a6ef55f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:32.879061 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:32.879034 5111 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:39:32.879061 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:32.879042 5111 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85444c64c6-qk7p8: secret "image-registry-tls" not found Nov 03 19:39:32.879206 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:32.879067 5111 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls podName:ac78f686-0d5a-4ff1-8a04-101e5794b7d7 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:34.879056913 +0000 UTC m=+4.841435330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls") pod "image-registry-85444c64c6-qk7p8" (UID: "ac78f686-0d5a-4ff1-8a04-101e5794b7d7") : secret "image-registry-tls" not found Nov 03 19:39:33.280886 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:33.280801 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z9svh" event={"ID":"2dcde652-ceea-4a58-9746-eb2087dea702","Type":"ContainerStarted","Data":"9904ff7332092b184d40d162c74228275398ef822deb0108f7f8c567855f66ed"} Nov 03 19:39:34.286044 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:34.285995 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-sk4zs" event={"ID":"b2503201-686a-4cf9-b5a1-977c87e6b717","Type":"ContainerStarted","Data":"73ec595329bf3d885da7655acffb5b4ec5f2307d6343640fb3809d8bbc446397"} Nov 03 19:39:34.288194 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:34.288160 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" event={"ID":"1b0776e1-a654-4e61-ba94-6d81ddfc8877","Type":"ContainerStarted","Data":"d3692cb5fab4f27a617ac0f26814a3d82a7e83b34fe60377f56e8a47612cfbc4"} Nov 03 19:39:34.288314 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:34.288221 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-szqck" event={"ID":"1b0776e1-a654-4e61-ba94-6d81ddfc8877","Type":"ContainerStarted","Data":"419ff66e637b685f93dbdf9eb31b4fa3e14eec7fa9526c395b351b47e693d97b"} Nov 03 19:39:34.289977 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:34.289952 5111 generic.go:356] "Generic (PLEG): container finished" podID="69c72eb2-3839-4483-82e8-b7d0bf82c57e" containerID="d2ce890fb0dbd1dc435e63386ece22b4c82ba8b7f034c3c18d6aed9c9e152512" exitCode=0 Nov 03 19:39:34.290082 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:34.290027 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" event={"ID":"69c72eb2-3839-4483-82e8-b7d0bf82c57e","Type":"ContainerDied","Data":"d2ce890fb0dbd1dc435e63386ece22b4c82ba8b7f034c3c18d6aed9c9e152512"} Nov 03 19:39:34.290327 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:34.290309 5111 scope.go:122] "RemoveContainer" containerID="d2ce890fb0dbd1dc435e63386ece22b4c82ba8b7f034c3c18d6aed9c9e152512" Nov 03 19:39:34.291975 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:34.291951 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z9svh" event={"ID":"2dcde652-ceea-4a58-9746-eb2087dea702","Type":"ContainerStarted","Data":"3c016f1ae56fa3bffef6972cc4ff96cd9e35bf63e32a3ab648ece7693bb17838"} Nov 03 19:39:34.292054 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:34.291993 5111 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:39:34.901059 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:34.901022 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:34.901254 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:34.901079 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:34.901254 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:34.901113 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:34.901254 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:34.901242 5111 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:39:34.901363 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:39:34.901316 5111 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls podName:d134b335-9e0e-41c9-a676-566a6ef55f5a nodeName:}" failed. No retries permitted until 2026-11-03 19:39:38.901297719 +0000 UTC m=+8.863676151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-md5d2" (UID: "d134b335-9e0e-41c9-a676-566a6ef55f5a") : secret "insights-runtime-extractor-tls" not found Nov 03 19:39:34.903507 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:34.903479 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls\") pod \"image-registry-85444c64c6-qk7p8\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:34.903661 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:34.903642 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls\") pod \"image-registry-b48f6466f-rvj5f\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:35.039148 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:35.039103 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:35.049341 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:35.049315 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:35.298053 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:35.297971 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bqmb8" event={"ID":"69c72eb2-3839-4483-82e8-b7d0bf82c57e","Type":"ContainerStarted","Data":"2bb81e76a6f4ca582875f93bba3b9a001943950736a7ec81dd724d93df14d9f2"} Nov 03 19:39:35.300513 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:35.300490 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zdf6j" event={"ID":"5101a457-b490-4538-a9f6-288014fa9264","Type":"ContainerStarted","Data":"87f42bff6bfd58f5fab4844d7aae2b299e5614b1c399859d148c541e3777f241"} Nov 03 19:39:35.300625 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:35.300544 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zdf6j" event={"ID":"5101a457-b490-4538-a9f6-288014fa9264","Type":"ContainerStarted","Data":"539931f2ed3d680ebc64160e439e8ce0a4d787ae29417c9edb441d24d436f828"} Nov 03 19:39:35.319431 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:35.319405 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85444c64c6-qk7p8"] Nov 03 19:39:35.322718 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:35.322690 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac78f686_0d5a_4ff1_8a04_101e5794b7d7.slice/crio-c8181afa964d7d9105801bdd962a2cdb4813b4b32b15849052ac7b2afa1ba3bf WatchSource:0}: Error finding container c8181afa964d7d9105801bdd962a2cdb4813b4b32b15849052ac7b2afa1ba3bf: Status 404 returned error can't find the container with id c8181afa964d7d9105801bdd962a2cdb4813b4b32b15849052ac7b2afa1ba3bf Nov 03 19:39:35.332138 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:35.332114 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b48f6466f-rvj5f"] Nov 03 19:39:35.335472 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:35.335451 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bab2991_93a7_44c8_9071_e3c971b79885.slice/crio-bfd8d0b5850e254374d1ff7fedd661eead41771235e10985adb1b2c459f67898 WatchSource:0}: Error finding container bfd8d0b5850e254374d1ff7fedd661eead41771235e10985adb1b2c459f67898: Status 404 returned error can't find the container with id bfd8d0b5850e254374d1ff7fedd661eead41771235e10985adb1b2c459f67898 Nov 03 19:39:36.305064 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:36.305021 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" event={"ID":"ac78f686-0d5a-4ff1-8a04-101e5794b7d7","Type":"ContainerStarted","Data":"0a776d133e583fdc4f3979f02f0f4969e65e28d0ff8dad781dba98fcaddbacf1"} Nov 03 19:39:36.305553 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:36.305092 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" event={"ID":"ac78f686-0d5a-4ff1-8a04-101e5794b7d7","Type":"ContainerStarted","Data":"c8181afa964d7d9105801bdd962a2cdb4813b4b32b15849052ac7b2afa1ba3bf"} Nov 03 19:39:36.306520 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:36.306492 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" event={"ID":"6bab2991-93a7-44c8-9071-e3c971b79885","Type":"ContainerStarted","Data":"b97a19cb3aa7f8d85f3998859164cce9d7799badd17de568359a47a10e6de8cc"} Nov 03 19:39:36.306643 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:36.306558 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" event={"ID":"6bab2991-93a7-44c8-9071-e3c971b79885","Type":"ContainerStarted","Data":"bfd8d0b5850e254374d1ff7fedd661eead41771235e10985adb1b2c459f67898"} Nov 03 19:39:36.306873 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:36.306739 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:37.311761 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:37.311729 5111 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:39:37.311761 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:37.311730 5111 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:39:37.440417 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:37.440387 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:38.571912 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:38.571885 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:38.572351 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:38.572124 5111 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:39:38.938895 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:38.938858 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:38.941176 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:38.941150 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d134b335-9e0e-41c9-a676-566a6ef55f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-md5d2\" (UID: \"d134b335-9e0e-41c9-a676-566a6ef55f5a\") " pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:38.957157 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:38.957131 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-md5d2" Nov 03 19:39:39.042002 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:39.041967 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:39.042127 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:39.042101 5111 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:39:39.045680 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:39.045659 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-68j2z" Nov 03 19:39:39.235167 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:39.235139 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-md5d2"] Nov 03 19:39:39.237727 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:39.237701 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd134b335_9e0e_41c9_a676_566a6ef55f5a.slice/crio-c2026239009e2b038bf8deaa55cf8d5ab2af449d02cf60cf1ba948ab20a0cd92 WatchSource:0}: Error finding container c2026239009e2b038bf8deaa55cf8d5ab2af449d02cf60cf1ba948ab20a0cd92: Status 404 returned error can't find the container with id c2026239009e2b038bf8deaa55cf8d5ab2af449d02cf60cf1ba948ab20a0cd92 Nov 03 19:39:39.320449 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:39.320417 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-md5d2" event={"ID":"d134b335-9e0e-41c9-a676-566a6ef55f5a","Type":"ContainerStarted","Data":"56ea2a2cc0ef3b3a0749796a10b8ae4ca16e982a8d9073e833ac833e5a428681"} Nov 03 19:39:39.320600 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:39.320478 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-md5d2" event={"ID":"d134b335-9e0e-41c9-a676-566a6ef55f5a","Type":"ContainerStarted","Data":"c2026239009e2b038bf8deaa55cf8d5ab2af449d02cf60cf1ba948ab20a0cd92"} Nov 03 19:39:39.437126 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:39.437086 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:39:39.636523 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:39.636493 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:39.637001 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:39.636644 5111 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:39:39.637806 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:39.637785 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-855cd4cff-prjqj" Nov 03 19:39:40.325437 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:40.325363 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-md5d2" event={"ID":"d134b335-9e0e-41c9-a676-566a6ef55f5a","Type":"ContainerStarted","Data":"9e504bd13c43f6e5a4d4fe5d69b15962f7319a5eaa04ca76b54cf596cdaaee0b"} Nov 03 19:39:42.334949 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:42.334909 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-md5d2" event={"ID":"d134b335-9e0e-41c9-a676-566a6ef55f5a","Type":"ContainerStarted","Data":"c10edb2595374599097735d48ce6925421a56e67f1c0a4775edc02bff37384e9"} Nov 03 19:39:43.199214 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:43.199183 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:46.246944 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.246913 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58c5cc9f56-qwcsc"] Nov 03 19:39:46.250737 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.250710 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.255554 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.255509 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-6h5km"] Nov 03 19:39:46.259107 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.258790 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-6h5km" Nov 03 19:39:46.260304 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.260276 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 19:39:46.264175 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.264151 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85444c64c6-qk7p8"] Nov 03 19:39:46.264702 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.264669 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 19:39:46.264836 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.264810 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 19:39:46.264970 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.264954 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 19:39:46.265156 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.265011 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 19:39:46.265362 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.265344 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rs9bf"] Nov 03 19:39:46.268261 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.268236 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 19:39:46.270652 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.270634 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 19:39:46.271520 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.271449 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xw588\"" Nov 03 19:39:46.272228 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.272208 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 19:39:46.272354 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.272249 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-2srp2\"" Nov 03 19:39:46.273736 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.273717 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58c5cc9f56-qwcsc"] Nov 03 19:39:46.274270 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.273836 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rs9bf" Nov 03 19:39:46.282926 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.282899 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-6h5km"] Nov 03 19:39:46.284417 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.283773 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rs9bf"] Nov 03 19:39:46.314042 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.314020 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zdf6j" Nov 03 19:39:46.338939 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.338914 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c87ddc7f5-dqz46"] Nov 03 19:39:46.342214 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.342198 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.356387 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.356366 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c87ddc7f5-dqz46"] Nov 03 19:39:46.370629 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.370606 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nztv8\"" Nov 03 19:39:46.370738 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.370694 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 19:39:46.403797 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.403764 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-oauth-config\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.403953 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.403831 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-trusted-ca-bundle\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.403953 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.403884 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/98e50f06-c94a-4d3c-a291-a85f0d18e564-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rs9bf\" (UID: \"98e50f06-c94a-4d3c-a291-a85f0d18e564\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rs9bf" Nov 03 19:39:46.403953 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.403932 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-service-ca\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.404126 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.403961 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-serving-cert\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.404126 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.403994 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-oauth-serving-cert\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.404126 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.404023 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1739913f-6ecd-4a90-80c6-ac0c7c36d224-tmp\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.404284 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.404152 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-config\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.404284 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.404241 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5211dbec-f5a3-4b26-ba64-171cb475f97c-tmp\") pod \"downloads-6dd789c8b7-6h5km\" (UID: \"5211dbec-f5a3-4b26-ba64-171cb475f97c\") " pod="openshift-console/downloads-6dd789c8b7-6h5km" Nov 03 19:39:46.404284 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.404270 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4d4p\" (UniqueName: \"kubernetes.io/projected/1739913f-6ecd-4a90-80c6-ac0c7c36d224-kube-api-access-n4d4p\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.404445 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.404292 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrrhj\" (UniqueName: \"kubernetes.io/projected/5211dbec-f5a3-4b26-ba64-171cb475f97c-kube-api-access-wrrhj\") pod \"downloads-6dd789c8b7-6h5km\" (UID: \"5211dbec-f5a3-4b26-ba64-171cb475f97c\") " pod="openshift-console/downloads-6dd789c8b7-6h5km" Nov 03 19:39:46.505693 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.505621 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-installation-pull-secrets\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.505693 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.505667 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/98e50f06-c94a-4d3c-a291-a85f0d18e564-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rs9bf\" (UID: \"98e50f06-c94a-4d3c-a291-a85f0d18e564\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rs9bf" Nov 03 19:39:46.505887 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.505704 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlwfl\" (UniqueName: \"kubernetes.io/projected/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-kube-api-access-rlwfl\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.505887 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.505854 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-service-ca\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.505887 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.505881 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-registry-tls\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.506005 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.505914 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-serving-cert\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.506005 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.505942 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-image-registry-private-configuration\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.506107 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506028 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-oauth-serving-cert\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.506107 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506056 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-trusted-ca\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.506107 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506078 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1739913f-6ecd-4a90-80c6-ac0c7c36d224-tmp\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.506107 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506103 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-config\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.506364 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506160 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5211dbec-f5a3-4b26-ba64-171cb475f97c-tmp\") pod \"downloads-6dd789c8b7-6h5km\" (UID: \"5211dbec-f5a3-4b26-ba64-171cb475f97c\") " pod="openshift-console/downloads-6dd789c8b7-6h5km" Nov 03 19:39:46.506364 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506182 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4d4p\" (UniqueName: \"kubernetes.io/projected/1739913f-6ecd-4a90-80c6-ac0c7c36d224-kube-api-access-n4d4p\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.506364 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506204 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrrhj\" (UniqueName: \"kubernetes.io/projected/5211dbec-f5a3-4b26-ba64-171cb475f97c-kube-api-access-wrrhj\") pod \"downloads-6dd789c8b7-6h5km\" (UID: \"5211dbec-f5a3-4b26-ba64-171cb475f97c\") " pod="openshift-console/downloads-6dd789c8b7-6h5km" Nov 03 19:39:46.506364 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506254 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-oauth-config\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.506364 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506293 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-registry-certificates\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.506364 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506323 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-ca-trust-extracted\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.506364 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506352 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-trusted-ca-bundle\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.506774 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506378 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-bound-sa-token\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.506774 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506581 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1739913f-6ecd-4a90-80c6-ac0c7c36d224-tmp\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.506875 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506784 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-service-ca\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.506875 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506816 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5211dbec-f5a3-4b26-ba64-171cb475f97c-tmp\") pod \"downloads-6dd789c8b7-6h5km\" (UID: \"5211dbec-f5a3-4b26-ba64-171cb475f97c\") " pod="openshift-console/downloads-6dd789c8b7-6h5km" Nov 03 19:39:46.506875 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506848 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-oauth-serving-cert\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.507014 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.506872 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-config\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.508056 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.508030 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-trusted-ca-bundle\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.508417 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.508396 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/98e50f06-c94a-4d3c-a291-a85f0d18e564-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rs9bf\" (UID: \"98e50f06-c94a-4d3c-a291-a85f0d18e564\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rs9bf" Nov 03 19:39:46.508673 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.508652 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-serving-cert\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.509224 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.509203 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-oauth-config\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.515219 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.515192 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrrhj\" (UniqueName: \"kubernetes.io/projected/5211dbec-f5a3-4b26-ba64-171cb475f97c-kube-api-access-wrrhj\") pod \"downloads-6dd789c8b7-6h5km\" (UID: \"5211dbec-f5a3-4b26-ba64-171cb475f97c\") " pod="openshift-console/downloads-6dd789c8b7-6h5km" Nov 03 19:39:46.515426 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.515404 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4d4p\" (UniqueName: \"kubernetes.io/projected/1739913f-6ecd-4a90-80c6-ac0c7c36d224-kube-api-access-n4d4p\") pod \"console-58c5cc9f56-qwcsc\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.564590 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.564556 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:46.573810 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.573786 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-6h5km" Nov 03 19:39:46.592445 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.592398 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rs9bf" Nov 03 19:39:46.606942 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.606911 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-registry-certificates\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.607058 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.606947 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-ca-trust-extracted\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.607058 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.606967 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-bound-sa-token\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.607058 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.606991 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-installation-pull-secrets\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.607430 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.607407 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-ca-trust-extracted\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.607524 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.607466 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlwfl\" (UniqueName: \"kubernetes.io/projected/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-kube-api-access-rlwfl\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.607524 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.607514 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-registry-tls\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.607649 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.607565 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-image-registry-private-configuration\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.607649 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.607592 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-trusted-ca\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.608719 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.608689 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-trusted-ca\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.608808 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.608714 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-registry-certificates\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.611022 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.610974 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-registry-tls\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.611248 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.611151 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-installation-pull-secrets\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.611248 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.611228 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-image-registry-private-configuration\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.618468 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.618418 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-bound-sa-token\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.618978 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.618959 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlwfl\" (UniqueName: \"kubernetes.io/projected/a91cdbfb-3a68-43c4-b773-6a8e2f32d224-kube-api-access-rlwfl\") pod \"image-registry-7c87ddc7f5-dqz46\" (UID: \"a91cdbfb-3a68-43c4-b773-6a8e2f32d224\") " pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.654520 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.654245 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:46.852681 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.852458 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58c5cc9f56-qwcsc"] Nov 03 19:39:46.855413 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:46.855377 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1739913f_6ecd_4a90_80c6_ac0c7c36d224.slice/crio-14717762bb24715332126bd006de1121fe8ee89e5d5421ba262afb0f916db79c WatchSource:0}: Error finding container 14717762bb24715332126bd006de1121fe8ee89e5d5421ba262afb0f916db79c: Status 404 returned error can't find the container with id 14717762bb24715332126bd006de1121fe8ee89e5d5421ba262afb0f916db79c Nov 03 19:39:46.855697 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.855677 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-6h5km"] Nov 03 19:39:46.859763 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:46.859734 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5211dbec_f5a3_4b26_ba64_171cb475f97c.slice/crio-fea027e6fd23a9c9540b8e420da02270d008d15af519865ff4dae77c236ea307 WatchSource:0}: Error finding container fea027e6fd23a9c9540b8e420da02270d008d15af519865ff4dae77c236ea307: Status 404 returned error can't find the container with id fea027e6fd23a9c9540b8e420da02270d008d15af519865ff4dae77c236ea307 Nov 03 19:39:46.879562 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.879524 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rs9bf"] Nov 03 19:39:46.882179 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:46.882156 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e50f06_c94a_4d3c_a291_a85f0d18e564.slice/crio-3d89f429a28573466ec9d04b5f7ca49fc7a5687c19df4527a988edb746282db3 WatchSource:0}: Error finding container 3d89f429a28573466ec9d04b5f7ca49fc7a5687c19df4527a988edb746282db3: Status 404 returned error can't find the container with id 3d89f429a28573466ec9d04b5f7ca49fc7a5687c19df4527a988edb746282db3 Nov 03 19:39:46.942429 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:46.942403 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c87ddc7f5-dqz46"] Nov 03 19:39:46.945562 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:46.945512 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91cdbfb_3a68_43c4_b773_6a8e2f32d224.slice/crio-e9c9b129dd7ffcb8166f4afb1e2de8d0d490eca82dcde16c2c21b095537cf99c WatchSource:0}: Error finding container e9c9b129dd7ffcb8166f4afb1e2de8d0d490eca82dcde16c2c21b095537cf99c: Status 404 returned error can't find the container with id e9c9b129dd7ffcb8166f4afb1e2de8d0d490eca82dcde16c2c21b095537cf99c Nov 03 19:39:47.361993 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:47.361951 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" event={"ID":"a91cdbfb-3a68-43c4-b773-6a8e2f32d224","Type":"ContainerStarted","Data":"38fc0c9219e7933171bd36d3e2f860c397ed0f0bb792f8d5b023aced2660dec4"} Nov 03 19:39:47.362449 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:47.362024 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" event={"ID":"a91cdbfb-3a68-43c4-b773-6a8e2f32d224","Type":"ContainerStarted","Data":"e9c9b129dd7ffcb8166f4afb1e2de8d0d490eca82dcde16c2c21b095537cf99c"} Nov 03 19:39:47.362449 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:47.362068 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:39:47.363222 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:47.363202 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-6h5km" event={"ID":"5211dbec-f5a3-4b26-ba64-171cb475f97c","Type":"ContainerStarted","Data":"fea027e6fd23a9c9540b8e420da02270d008d15af519865ff4dae77c236ea307"} Nov 03 19:39:47.364353 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:47.364333 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c5cc9f56-qwcsc" event={"ID":"1739913f-6ecd-4a90-80c6-ac0c7c36d224","Type":"ContainerStarted","Data":"14717762bb24715332126bd006de1121fe8ee89e5d5421ba262afb0f916db79c"} Nov 03 19:39:47.365417 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:47.365396 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rs9bf" event={"ID":"98e50f06-c94a-4d3c-a291-a85f0d18e564","Type":"ContainerStarted","Data":"3d89f429a28573466ec9d04b5f7ca49fc7a5687c19df4527a988edb746282db3"} Nov 03 19:39:47.381937 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:47.381883 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" podStartSLOduration=1.381872099 podStartE2EDuration="1.381872099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:47.380887328 +0000 UTC m=+17.343265769" watchObservedRunningTime="2026-11-03 19:39:47.381872099 +0000 UTC m=+17.344250537" Nov 03 19:39:48.373009 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:48.372966 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rs9bf" event={"ID":"98e50f06-c94a-4d3c-a291-a85f0d18e564","Type":"ContainerStarted","Data":"671d57bbe0a19fd50885616ecfbc1257a14539d70169f304ea63aa5cd26bf430"} Nov 03 19:39:48.373621 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:48.373212 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rs9bf" Nov 03 19:39:48.378656 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:48.378633 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rs9bf" Nov 03 19:39:48.389708 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:48.389616 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rs9bf" podStartSLOduration=1.148036557 podStartE2EDuration="2.389600079s" totalImagesPullingTime="1.241563522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:48.388455867 +0000 UTC m=+18.350834305" watchObservedRunningTime="2026-11-03 19:39:48.389600079 +0000 UTC m=+18.351978514" Nov 03 19:39:50.043709 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:50.043679 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzzd5" Nov 03 19:39:50.390023 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:50.389985 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c5cc9f56-qwcsc" event={"ID":"1739913f-6ecd-4a90-80c6-ac0c7c36d224","Type":"ContainerStarted","Data":"0d38791f1d88d628b4d854db5ea2458362b90f7c6369d8996336597e236ef9e3"} Nov 03 19:39:50.410361 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:50.410311 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58c5cc9f56-qwcsc" podStartSLOduration=1.139369966 podStartE2EDuration="4.410299869s" totalImagesPullingTime="3.270929903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:50.409738766 +0000 UTC m=+20.372117207" watchObservedRunningTime="2026-11-03 19:39:50.410299869 +0000 UTC m=+20.372678305" Nov 03 19:39:52.908108 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:52.908077 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8sczc"] Nov 03 19:39:52.922574 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:52.917097 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:52.935002 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:52.933976 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g7dvr\"" Nov 03 19:39:52.935002 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:52.934069 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 19:39:52.935002 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:52.934175 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 19:39:53.017094 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.017066 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 19:39:53.017094 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.017093 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 19:39:53.073719 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.073680 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8gd9\" (UniqueName: \"kubernetes.io/projected/83bba5dc-5ae2-4fab-a021-d828309dd462-kube-api-access-j8gd9\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.073880 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.073739 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/83bba5dc-5ae2-4fab-a021-d828309dd462-node-exporter-accelerators-collector-config\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.073939 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.073881 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83bba5dc-5ae2-4fab-a021-d828309dd462-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.073939 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.073917 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/83bba5dc-5ae2-4fab-a021-d828309dd462-node-exporter-tls\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.074015 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.073986 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/83bba5dc-5ae2-4fab-a021-d828309dd462-root\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.074065 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.074047 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83bba5dc-5ae2-4fab-a021-d828309dd462-sys\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.074124 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.074097 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/83bba5dc-5ae2-4fab-a021-d828309dd462-node-exporter-textfile\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.074172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.074128 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83bba5dc-5ae2-4fab-a021-d828309dd462-metrics-client-ca\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.074215 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.074203 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/83bba5dc-5ae2-4fab-a021-d828309dd462-node-exporter-wtmp\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.175573 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.175475 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8gd9\" (UniqueName: \"kubernetes.io/projected/83bba5dc-5ae2-4fab-a021-d828309dd462-kube-api-access-j8gd9\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.175573 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.175569 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/83bba5dc-5ae2-4fab-a021-d828309dd462-node-exporter-accelerators-collector-config\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.175830 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.175620 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83bba5dc-5ae2-4fab-a021-d828309dd462-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.175830 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.175654 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/83bba5dc-5ae2-4fab-a021-d828309dd462-node-exporter-tls\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.175830 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.175697 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/83bba5dc-5ae2-4fab-a021-d828309dd462-root\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.175830 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.175723 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83bba5dc-5ae2-4fab-a021-d828309dd462-sys\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.175830 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.175740 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/83bba5dc-5ae2-4fab-a021-d828309dd462-node-exporter-textfile\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.175830 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.175753 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83bba5dc-5ae2-4fab-a021-d828309dd462-metrics-client-ca\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.175830 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.175805 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/83bba5dc-5ae2-4fab-a021-d828309dd462-node-exporter-wtmp\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.176176 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.175911 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/83bba5dc-5ae2-4fab-a021-d828309dd462-root\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.176176 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.175943 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/83bba5dc-5ae2-4fab-a021-d828309dd462-node-exporter-wtmp\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.176284 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.176237 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/83bba5dc-5ae2-4fab-a021-d828309dd462-node-exporter-textfile\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.176337 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.176285 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/83bba5dc-5ae2-4fab-a021-d828309dd462-node-exporter-accelerators-collector-config\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.176448 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.176423 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83bba5dc-5ae2-4fab-a021-d828309dd462-sys\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.176687 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.176664 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83bba5dc-5ae2-4fab-a021-d828309dd462-metrics-client-ca\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.178954 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.178909 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/83bba5dc-5ae2-4fab-a021-d828309dd462-node-exporter-tls\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.178954 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.178918 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83bba5dc-5ae2-4fab-a021-d828309dd462-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.184405 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.184386 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8gd9\" (UniqueName: \"kubernetes.io/projected/83bba5dc-5ae2-4fab-a021-d828309dd462-kube-api-access-j8gd9\") pod \"node-exporter-8sczc\" (UID: \"83bba5dc-5ae2-4fab-a021-d828309dd462\") " pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.200517 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.200486 5111 patch_prober.go:29] interesting pod/image-registry-b48f6466f-rvj5f container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 19:39:53.200667 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.200551 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" podUID="6bab2991-93a7-44c8-9071-e3c971b79885" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:39:53.250050 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.249831 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8sczc" Nov 03 19:39:53.265693 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:39:53.265660 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bba5dc_5ae2_4fab_a021_d828309dd462.slice/crio-16c27b3ac68b0dd28ea9ba0889cb3798bdba9431a634a48099255af9bc1f3622 WatchSource:0}: Error finding container 16c27b3ac68b0dd28ea9ba0889cb3798bdba9431a634a48099255af9bc1f3622: Status 404 returned error can't find the container with id 16c27b3ac68b0dd28ea9ba0889cb3798bdba9431a634a48099255af9bc1f3622 Nov 03 19:39:53.406442 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:53.406397 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8sczc" event={"ID":"83bba5dc-5ae2-4fab-a021-d828309dd462","Type":"ContainerStarted","Data":"16c27b3ac68b0dd28ea9ba0889cb3798bdba9431a634a48099255af9bc1f3622"} Nov 03 19:39:54.733363 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:54.733326 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f4x7r" Nov 03 19:39:55.428503 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:55.428468 5111 generic.go:356] "Generic (PLEG): container finished" podID="83bba5dc-5ae2-4fab-a021-d828309dd462" containerID="13155ad7faee69b608394a4defe846d3488f381ba45b6c661cd4007787b671e3" exitCode=0 Nov 03 19:39:55.428713 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:55.428521 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8sczc" event={"ID":"83bba5dc-5ae2-4fab-a021-d828309dd462","Type":"ContainerDied","Data":"13155ad7faee69b608394a4defe846d3488f381ba45b6c661cd4007787b671e3"} Nov 03 19:39:56.271213 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:56.271182 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:39:56.435426 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:56.435394 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8sczc" event={"ID":"83bba5dc-5ae2-4fab-a021-d828309dd462","Type":"ContainerStarted","Data":"bc8edb6fce42139bf84dd096a5af7ed5327b827db42ac0a4b20820143663b8e9"} Nov 03 19:39:56.435610 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:56.435453 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8sczc" event={"ID":"83bba5dc-5ae2-4fab-a021-d828309dd462","Type":"ContainerStarted","Data":"b7ac90b6e6994f00bb3a8dd0c483e68ee1d64cd63bd8aa64edb8d48c286ca378"} Nov 03 19:39:56.455454 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:56.455404 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8sczc" podStartSLOduration=2.710996198 podStartE2EDuration="4.455390507s" totalImagesPullingTime="1.655615312s" totalInitContainerRuntime="88.778997ms" isStatefulPod=true podCreationTimestamp="2026-11-03 19:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:56.454233541 +0000 UTC m=+26.416611981" watchObservedRunningTime="2026-11-03 19:39:56.455390507 +0000 UTC m=+26.417768946" Nov 03 19:39:56.565211 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:56.565125 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:56.565965 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:56.565930 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:56.570239 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:56.570215 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:57.445188 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:57.445157 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:39:59.442160 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:39:59.442126 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:40:02.384164 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:02.384125 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wztqw" Nov 03 19:40:04.479773 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:04.479733 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-6h5km" event={"ID":"5211dbec-f5a3-4b26-ba64-171cb475f97c","Type":"ContainerStarted","Data":"400713ca28780f7c42e3a279eb95b810f9ff2236e40651c52419aeab415235bb"} Nov 03 19:40:04.480303 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:04.479963 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-6h5km" Nov 03 19:40:04.496545 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:04.496507 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-6h5km" Nov 03 19:40:04.500438 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:04.500385 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-6h5km" podStartSLOduration=1.631761491 podStartE2EDuration="18.500370475s" totalImagesPullingTime="16.868608984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:04.499340605 +0000 UTC m=+34.461719043" watchObservedRunningTime="2026-11-03 19:40:04.500370475 +0000 UTC m=+34.462748911" Nov 03 19:40:05.066060 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:40:05.066017 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:40:05.674199 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.674157 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-f9cc58d48-kmlkw"] Nov 03 19:40:05.720208 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.720169 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f9cc58d48-kmlkw"] Nov 03 19:40:05.720459 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.720238 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.792187 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.792142 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-oauth-config\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.792373 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.792213 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-oauth-serving-cert\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.792373 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.792349 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-trusted-ca-bundle\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.792479 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.792410 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-serving-cert\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.792479 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.792464 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgxsw\" (UniqueName: \"kubernetes.io/projected/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-kube-api-access-tgxsw\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.792614 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.792501 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-service-ca\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.792614 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.792540 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-tmp\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.792699 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.792660 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-config\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.893704 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.893661 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-config\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.893902 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.893723 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-oauth-config\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.893902 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.893766 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-oauth-serving-cert\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.893902 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.893811 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-trusted-ca-bundle\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.893902 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.893856 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-serving-cert\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.893902 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.893886 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgxsw\" (UniqueName: \"kubernetes.io/projected/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-kube-api-access-tgxsw\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.894393 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.894354 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-service-ca\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.895303 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.895272 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-tmp\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.895491 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.894557 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-config\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.895644 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.894853 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-trusted-ca-bundle\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.895742 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.895200 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-service-ca\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.895829 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.894548 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-oauth-serving-cert\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.895909 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.895580 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-tmp\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.897171 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.897149 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-serving-cert\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.897572 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.897548 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-oauth-config\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:05.903631 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:05.903609 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgxsw\" (UniqueName: \"kubernetes.io/projected/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-kube-api-access-tgxsw\") pod \"console-f9cc58d48-kmlkw\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:06.060848 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:06.060768 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:06.376995 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:06.376813 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f9cc58d48-kmlkw"] Nov 03 19:40:06.382904 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:40:06.382870 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2cb9475_747e_460b_b7fc_8c4bd6ff0b5b.slice/crio-2f98afaf5addf810ee7e05800655ed3c5bd96603e22d098e5c6420674f45e099 WatchSource:0}: Error finding container 2f98afaf5addf810ee7e05800655ed3c5bd96603e22d098e5c6420674f45e099: Status 404 returned error can't find the container with id 2f98afaf5addf810ee7e05800655ed3c5bd96603e22d098e5c6420674f45e099 Nov 03 19:40:06.492598 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:06.492555 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9cc58d48-kmlkw" event={"ID":"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b","Type":"ContainerStarted","Data":"9dfe8d059be6cc08cbdf23adf9fe700ed51ec1e246179474b4c163e401fc056e"} Nov 03 19:40:06.492797 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:06.492622 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9cc58d48-kmlkw" event={"ID":"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b","Type":"ContainerStarted","Data":"2f98afaf5addf810ee7e05800655ed3c5bd96603e22d098e5c6420674f45e099"} Nov 03 19:40:06.519280 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:06.519220 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-f9cc58d48-kmlkw" podStartSLOduration=1.5192011220000001 podStartE2EDuration="1.519201122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:06.517099067 +0000 UTC m=+36.479477504" watchObservedRunningTime="2026-11-03 19:40:06.519201122 +0000 UTC m=+36.481579560" Nov 03 19:40:08.378605 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:08.378556 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c87ddc7f5-dqz46" Nov 03 19:40:08.435233 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:08.435192 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b48f6466f-rvj5f"] Nov 03 19:40:11.292306 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.292194 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" podUID="ac78f686-0d5a-4ff1-8a04-101e5794b7d7" containerName="registry" containerID="cri-o://0a776d133e583fdc4f3979f02f0f4969e65e28d0ff8dad781dba98fcaddbacf1" gracePeriod=30 Nov 03 19:40:11.522512 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.522474 5111 generic.go:356] "Generic (PLEG): container finished" podID="ac78f686-0d5a-4ff1-8a04-101e5794b7d7" containerID="0a776d133e583fdc4f3979f02f0f4969e65e28d0ff8dad781dba98fcaddbacf1" exitCode=0 Nov 03 19:40:11.522760 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.522578 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" event={"ID":"ac78f686-0d5a-4ff1-8a04-101e5794b7d7","Type":"ContainerDied","Data":"0a776d133e583fdc4f3979f02f0f4969e65e28d0ff8dad781dba98fcaddbacf1"} Nov 03 19:40:11.594419 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.594381 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:40:11.654099 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.654067 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-bound-sa-token\") pod \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " Nov 03 19:40:11.654301 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.654126 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-trusted-ca\") pod \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " Nov 03 19:40:11.654301 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.654208 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-ca-trust-extracted\") pod \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " Nov 03 19:40:11.654301 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.654271 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-installation-pull-secrets\") pod \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " Nov 03 19:40:11.655076 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.654326 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls\") pod \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " Nov 03 19:40:11.655076 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.654406 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-certificates\") pod \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " Nov 03 19:40:11.655076 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.654436 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-image-registry-private-configuration\") pod \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " Nov 03 19:40:11.655076 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.654487 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-kube-api-access-g4hfr\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-kube-api-access-g4hfr\") pod \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\" (UID: \"ac78f686-0d5a-4ff1-8a04-101e5794b7d7\") " Nov 03 19:40:11.655076 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.654690 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-trusted-ca" pod "ac78f686-0d5a-4ff1-8a04-101e5794b7d7" (UID: "ac78f686-0d5a-4ff1-8a04-101e5794b7d7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:11.655076 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.654814 5111 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-trusted-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:11.655464 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.655141 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-certificates" pod "ac78f686-0d5a-4ff1-8a04-101e5794b7d7" (UID: "ac78f686-0d5a-4ff1-8a04-101e5794b7d7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:11.658146 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.658093 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-kube-api-access-g4hfr" pod "ac78f686-0d5a-4ff1-8a04-101e5794b7d7" (UID: "ac78f686-0d5a-4ff1-8a04-101e5794b7d7"). InnerVolumeSpecName "kube-api-access-g4hfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:11.658239 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.658210 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls" pod "ac78f686-0d5a-4ff1-8a04-101e5794b7d7" (UID: "ac78f686-0d5a-4ff1-8a04-101e5794b7d7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:11.659256 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.659183 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-installation-pull-secrets" pod "ac78f686-0d5a-4ff1-8a04-101e5794b7d7" (UID: "ac78f686-0d5a-4ff1-8a04-101e5794b7d7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:11.660500 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.660473 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-bound-sa-token" pod "ac78f686-0d5a-4ff1-8a04-101e5794b7d7" (UID: "ac78f686-0d5a-4ff1-8a04-101e5794b7d7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:11.660634 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.660497 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-image-registry-private-configuration" pod "ac78f686-0d5a-4ff1-8a04-101e5794b7d7" (UID: "ac78f686-0d5a-4ff1-8a04-101e5794b7d7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:11.663884 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.663838 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-ca-trust-extracted" pod "ac78f686-0d5a-4ff1-8a04-101e5794b7d7" (UID: "ac78f686-0d5a-4ff1-8a04-101e5794b7d7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:40:11.755591 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.755546 5111 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-bound-sa-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:11.755591 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.755594 5111 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-ca-trust-extracted\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:11.755800 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.755610 5111 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-installation-pull-secrets\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:11.755800 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.755628 5111 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-tls\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:11.755800 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.755642 5111 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-registry-certificates\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:11.755800 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.755657 5111 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-image-registry-private-configuration\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:11.755800 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:11.755673 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4hfr\" (UniqueName: \"kubernetes.io/projected/ac78f686-0d5a-4ff1-8a04-101e5794b7d7-kube-api-access-g4hfr\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:12.530068 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:12.529970 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" Nov 03 19:40:12.530068 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:12.529968 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85444c64c6-qk7p8" event={"ID":"ac78f686-0d5a-4ff1-8a04-101e5794b7d7","Type":"ContainerDied","Data":"c8181afa964d7d9105801bdd962a2cdb4813b4b32b15849052ac7b2afa1ba3bf"} Nov 03 19:40:12.530603 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:12.530095 5111 scope.go:122] "RemoveContainer" containerID="0a776d133e583fdc4f3979f02f0f4969e65e28d0ff8dad781dba98fcaddbacf1" Nov 03 19:40:12.548696 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:12.548666 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85444c64c6-qk7p8"] Nov 03 19:40:12.551069 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:12.551031 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85444c64c6-qk7p8"] Nov 03 19:40:14.180144 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:14.180110 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac78f686-0d5a-4ff1-8a04-101e5794b7d7" path="/var/lib/kubelet/pods/ac78f686-0d5a-4ff1-8a04-101e5794b7d7/volumes" Nov 03 19:40:16.061276 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:16.061236 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:16.061705 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:16.061322 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:16.066257 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:16.066229 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:16.552066 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:16.552037 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:40:16.596233 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:16.596202 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58c5cc9f56-qwcsc"] Nov 03 19:40:33.463224 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.463174 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" podUID="6bab2991-93a7-44c8-9071-e3c971b79885" containerName="registry" containerID="cri-o://b97a19cb3aa7f8d85f3998859164cce9d7799badd17de568359a47a10e6de8cc" gracePeriod=30 Nov 03 19:40:33.621988 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.621815 5111 generic.go:356] "Generic (PLEG): container finished" podID="6bab2991-93a7-44c8-9071-e3c971b79885" containerID="b97a19cb3aa7f8d85f3998859164cce9d7799badd17de568359a47a10e6de8cc" exitCode=0 Nov 03 19:40:33.621988 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.621932 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" event={"ID":"6bab2991-93a7-44c8-9071-e3c971b79885","Type":"ContainerDied","Data":"b97a19cb3aa7f8d85f3998859164cce9d7799badd17de568359a47a10e6de8cc"} Nov 03 19:40:33.744731 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.744706 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:40:33.845748 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.845710 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-trusted-ca\") pod \"6bab2991-93a7-44c8-9071-e3c971b79885\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " Nov 03 19:40:33.845915 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.845760 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-kube-api-access-c67h2\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-kube-api-access-c67h2\") pod \"6bab2991-93a7-44c8-9071-e3c971b79885\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " Nov 03 19:40:33.845915 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.845793 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-bound-sa-token\") pod \"6bab2991-93a7-44c8-9071-e3c971b79885\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " Nov 03 19:40:33.845915 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.845827 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-installation-pull-secrets\") pod \"6bab2991-93a7-44c8-9071-e3c971b79885\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " Nov 03 19:40:33.845915 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.845873 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-image-registry-private-configuration\") pod \"6bab2991-93a7-44c8-9071-e3c971b79885\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " Nov 03 19:40:33.846131 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.846046 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bab2991-93a7-44c8-9071-e3c971b79885-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6bab2991-93a7-44c8-9071-e3c971b79885-ca-trust-extracted\") pod \"6bab2991-93a7-44c8-9071-e3c971b79885\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " Nov 03 19:40:33.846131 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.846114 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-registry-certificates\") pod \"6bab2991-93a7-44c8-9071-e3c971b79885\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " Nov 03 19:40:33.846231 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.846163 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls\") pod \"6bab2991-93a7-44c8-9071-e3c971b79885\" (UID: \"6bab2991-93a7-44c8-9071-e3c971b79885\") " Nov 03 19:40:33.846231 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.846208 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-trusted-ca" pod "6bab2991-93a7-44c8-9071-e3c971b79885" (UID: "6bab2991-93a7-44c8-9071-e3c971b79885"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:33.846484 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.846464 5111 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-trusted-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:33.846597 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.846477 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-registry-certificates" pod "6bab2991-93a7-44c8-9071-e3c971b79885" (UID: "6bab2991-93a7-44c8-9071-e3c971b79885"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:33.848374 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.848348 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls" pod "6bab2991-93a7-44c8-9071-e3c971b79885" (UID: "6bab2991-93a7-44c8-9071-e3c971b79885"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:33.848482 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.848384 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-bound-sa-token" pod "6bab2991-93a7-44c8-9071-e3c971b79885" (UID: "6bab2991-93a7-44c8-9071-e3c971b79885"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:33.848566 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.848510 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-kube-api-access-c67h2" pod "6bab2991-93a7-44c8-9071-e3c971b79885" (UID: "6bab2991-93a7-44c8-9071-e3c971b79885"). InnerVolumeSpecName "kube-api-access-c67h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:33.848634 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.848616 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-image-registry-private-configuration" pod "6bab2991-93a7-44c8-9071-e3c971b79885" (UID: "6bab2991-93a7-44c8-9071-e3c971b79885"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:33.848732 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.848712 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-installation-pull-secrets" pod "6bab2991-93a7-44c8-9071-e3c971b79885" (UID: "6bab2991-93a7-44c8-9071-e3c971b79885"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:33.853285 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.853262 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bab2991-93a7-44c8-9071-e3c971b79885-ca-trust-extracted" pod "6bab2991-93a7-44c8-9071-e3c971b79885" (UID: "6bab2991-93a7-44c8-9071-e3c971b79885"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:40:33.946918 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.946881 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c67h2\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-kube-api-access-c67h2\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:33.946918 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.946912 5111 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-bound-sa-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:33.946918 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.946922 5111 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-installation-pull-secrets\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:33.947132 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.946933 5111 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6bab2991-93a7-44c8-9071-e3c971b79885-image-registry-private-configuration\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:33.947132 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.946944 5111 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6bab2991-93a7-44c8-9071-e3c971b79885-ca-trust-extracted\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:33.947132 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.946953 5111 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6bab2991-93a7-44c8-9071-e3c971b79885-registry-certificates\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:33.947132 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:33.946961 5111 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6bab2991-93a7-44c8-9071-e3c971b79885-registry-tls\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:34.627715 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:34.627679 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" event={"ID":"6bab2991-93a7-44c8-9071-e3c971b79885","Type":"ContainerDied","Data":"bfd8d0b5850e254374d1ff7fedd661eead41771235e10985adb1b2c459f67898"} Nov 03 19:40:34.627715 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:34.627714 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b48f6466f-rvj5f" Nov 03 19:40:34.628261 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:34.627739 5111 scope.go:122] "RemoveContainer" containerID="b97a19cb3aa7f8d85f3998859164cce9d7799badd17de568359a47a10e6de8cc" Nov 03 19:40:34.644633 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:34.644608 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b48f6466f-rvj5f"] Nov 03 19:40:34.648860 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:34.648837 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b48f6466f-rvj5f"] Nov 03 19:40:35.054502 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:40:35.054417 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:40:36.180945 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:36.180913 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bab2991-93a7-44c8-9071-e3c971b79885" path="/var/lib/kubelet/pods/6bab2991-93a7-44c8-9071-e3c971b79885/volumes" Nov 03 19:40:39.651463 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:39.651431 5111 generic.go:356] "Generic (PLEG): container finished" podID="2266dada-9604-44f9-b488-511006cee7b2" containerID="a5f6e3b2906b45fcc933c819d7b6e6f3a27347f3e9b920717d7c5b413bb181cd" exitCode=0 Nov 03 19:40:39.652055 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:39.651503 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rs8wk" event={"ID":"2266dada-9604-44f9-b488-511006cee7b2","Type":"ContainerDied","Data":"a5f6e3b2906b45fcc933c819d7b6e6f3a27347f3e9b920717d7c5b413bb181cd"} Nov 03 19:40:39.652055 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:39.651954 5111 scope.go:122] "RemoveContainer" containerID="a5f6e3b2906b45fcc933c819d7b6e6f3a27347f3e9b920717d7c5b413bb181cd" Nov 03 19:40:39.653118 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:39.653087 5111 generic.go:356] "Generic (PLEG): container finished" podID="74a9a94d-e73f-4942-bab6-efbfcb728df2" containerID="940af9c301b83dbaad9deab8529a9b1e3eaaf3716f05ccf096c0250869b9891e" exitCode=0 Nov 03 19:40:39.653231 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:39.653157 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" event={"ID":"74a9a94d-e73f-4942-bab6-efbfcb728df2","Type":"ContainerDied","Data":"940af9c301b83dbaad9deab8529a9b1e3eaaf3716f05ccf096c0250869b9891e"} Nov 03 19:40:39.653458 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:39.653441 5111 scope.go:122] "RemoveContainer" containerID="940af9c301b83dbaad9deab8529a9b1e3eaaf3716f05ccf096c0250869b9891e" Nov 03 19:40:40.031937 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.031852 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-fn7j5-kgmpq"] Nov 03 19:40:40.032086 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.032053 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" podUID="c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467" containerName="insights-gathering" containerID="cri-o://891f85911a5b5604b88db6cc45b9acc1fc4aab5697e26a3f2ca57a763135c4f3" gracePeriod=30 Nov 03 19:40:40.120349 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.120310 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-6x92p-7c5mm"] Nov 03 19:40:40.120751 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.120735 5111 cpu_manager.go:414] "containerMap: removing container" podUID="ac78f686-0d5a-4ff1-8a04-101e5794b7d7" containerName="registry" Nov 03 19:40:40.120810 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.120753 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac78f686-0d5a-4ff1-8a04-101e5794b7d7" containerName="registry" Nov 03 19:40:40.120810 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.120767 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6bab2991-93a7-44c8-9071-e3c971b79885" containerName="registry" Nov 03 19:40:40.120810 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.120773 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bab2991-93a7-44c8-9071-e3c971b79885" containerName="registry" Nov 03 19:40:40.120901 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.120839 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac78f686-0d5a-4ff1-8a04-101e5794b7d7" containerName="registry" Nov 03 19:40:40.120901 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.120847 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bab2991-93a7-44c8-9071-e3c971b79885" containerName="registry" Nov 03 19:40:40.139607 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.139573 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-6x92p-7c5mm"] Nov 03 19:40:40.139789 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.139664 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6x92p-7c5mm" Nov 03 19:40:40.204012 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.203978 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-service-ca-bundle\") pod \"periodic-gathering-6x92p-7c5mm\" (UID: \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\") " pod="openshift-insights/periodic-gathering-6x92p-7c5mm" Nov 03 19:40:40.204012 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.204014 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-archives-path\") pod \"periodic-gathering-6x92p-7c5mm\" (UID: \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\") " pod="openshift-insights/periodic-gathering-6x92p-7c5mm" Nov 03 19:40:40.204207 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.204045 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-trusted-ca-bundle\") pod \"periodic-gathering-6x92p-7c5mm\" (UID: \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\") " pod="openshift-insights/periodic-gathering-6x92p-7c5mm" Nov 03 19:40:40.204244 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.204221 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4fc2\" (UniqueName: \"kubernetes.io/projected/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-kube-api-access-v4fc2\") pod \"periodic-gathering-6x92p-7c5mm\" (UID: \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\") " pod="openshift-insights/periodic-gathering-6x92p-7c5mm" Nov 03 19:40:40.305256 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.305225 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4fc2\" (UniqueName: \"kubernetes.io/projected/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-kube-api-access-v4fc2\") pod \"periodic-gathering-6x92p-7c5mm\" (UID: \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\") " pod="openshift-insights/periodic-gathering-6x92p-7c5mm" Nov 03 19:40:40.305404 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.305271 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-service-ca-bundle\") pod \"periodic-gathering-6x92p-7c5mm\" (UID: \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\") " pod="openshift-insights/periodic-gathering-6x92p-7c5mm" Nov 03 19:40:40.305404 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.305300 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-archives-path\") pod \"periodic-gathering-6x92p-7c5mm\" (UID: \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\") " pod="openshift-insights/periodic-gathering-6x92p-7c5mm" Nov 03 19:40:40.305404 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.305327 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-trusted-ca-bundle\") pod \"periodic-gathering-6x92p-7c5mm\" (UID: \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\") " pod="openshift-insights/periodic-gathering-6x92p-7c5mm" Nov 03 19:40:40.305678 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.305658 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-archives-path\") pod \"periodic-gathering-6x92p-7c5mm\" (UID: \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\") " pod="openshift-insights/periodic-gathering-6x92p-7c5mm" Nov 03 19:40:40.305892 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.305875 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-service-ca-bundle\") pod \"periodic-gathering-6x92p-7c5mm\" (UID: \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\") " pod="openshift-insights/periodic-gathering-6x92p-7c5mm" Nov 03 19:40:40.306323 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.306305 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-trusted-ca-bundle\") pod \"periodic-gathering-6x92p-7c5mm\" (UID: \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\") " pod="openshift-insights/periodic-gathering-6x92p-7c5mm" Nov 03 19:40:40.314029 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.314000 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4fc2\" (UniqueName: \"kubernetes.io/projected/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-kube-api-access-v4fc2\") pod \"periodic-gathering-6x92p-7c5mm\" (UID: \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\") " pod="openshift-insights/periodic-gathering-6x92p-7c5mm" Nov 03 19:40:40.327984 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.327963 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:40:40.406485 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.406447 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-kube-api-access-7zcvd\" (UniqueName: \"kubernetes.io/projected/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-kube-api-access-7zcvd\") pod \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " Nov 03 19:40:40.406708 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.406506 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-archives-path\") pod \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " Nov 03 19:40:40.406708 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.406556 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-service-ca-bundle\") pod \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " Nov 03 19:40:40.406708 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.406587 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-trusted-ca-bundle\") pod \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\" (UID: \"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467\") " Nov 03 19:40:40.406917 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.406890 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-archives-path" pod "c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467" (UID: "c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:40:40.406972 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.406921 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-service-ca-bundle" pod "c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467" (UID: "c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:40.407012 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.406973 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-trusted-ca-bundle" pod "c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467" (UID: "c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:40.408883 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.408856 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-kube-api-access-7zcvd" pod "c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467" (UID: "c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467"). InnerVolumeSpecName "kube-api-access-7zcvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:40.470126 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.470085 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6x92p-7c5mm" Nov 03 19:40:40.507495 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.507459 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zcvd\" (UniqueName: \"kubernetes.io/projected/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-kube-api-access-7zcvd\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:40.507495 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.507492 5111 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-archives-path\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:40.507708 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.507505 5111 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-service-ca-bundle\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:40.507708 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.507517 5111 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467-trusted-ca-bundle\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:40.658836 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.658800 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rs8wk" event={"ID":"2266dada-9604-44f9-b488-511006cee7b2","Type":"ContainerStarted","Data":"a0bda7e6bd6fc717e0b676dd0994f744178aa8abe00e24f57cdb9b749ceaeee0"} Nov 03 19:40:40.660626 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.660600 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-f5m2p" event={"ID":"74a9a94d-e73f-4942-bab6-efbfcb728df2","Type":"ContainerStarted","Data":"dbdff4fa312d853706c7eda0ad8725888b9d539935878d76d9fb6b0fecd2214e"} Nov 03 19:40:40.661685 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.661663 5111 generic.go:356] "Generic (PLEG): container finished" podID="c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467" containerID="891f85911a5b5604b88db6cc45b9acc1fc4aab5697e26a3f2ca57a763135c4f3" exitCode=2 Nov 03 19:40:40.661782 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.661696 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" Nov 03 19:40:40.661782 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.661724 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" event={"ID":"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467","Type":"ContainerDied","Data":"891f85911a5b5604b88db6cc45b9acc1fc4aab5697e26a3f2ca57a763135c4f3"} Nov 03 19:40:40.661782 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.661777 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fn7j5-kgmpq" event={"ID":"c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467","Type":"ContainerDied","Data":"5a8d093e45f6dd57c6ae8973495db5795a1a62dded9a37a264bc8eb64d3b0ee7"} Nov 03 19:40:40.661946 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.661805 5111 scope.go:122] "RemoveContainer" containerID="891f85911a5b5604b88db6cc45b9acc1fc4aab5697e26a3f2ca57a763135c4f3" Nov 03 19:40:40.674743 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.674723 5111 scope.go:122] "RemoveContainer" containerID="891f85911a5b5604b88db6cc45b9acc1fc4aab5697e26a3f2ca57a763135c4f3" Nov 03 19:40:40.675031 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:40:40.675009 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891f85911a5b5604b88db6cc45b9acc1fc4aab5697e26a3f2ca57a763135c4f3\": container with ID starting with 891f85911a5b5604b88db6cc45b9acc1fc4aab5697e26a3f2ca57a763135c4f3 not found: ID does not exist" containerID="891f85911a5b5604b88db6cc45b9acc1fc4aab5697e26a3f2ca57a763135c4f3" Nov 03 19:40:40.675093 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.675038 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891f85911a5b5604b88db6cc45b9acc1fc4aab5697e26a3f2ca57a763135c4f3"} err="failed to get container status \"891f85911a5b5604b88db6cc45b9acc1fc4aab5697e26a3f2ca57a763135c4f3\": rpc error: code = NotFound desc = could not find container \"891f85911a5b5604b88db6cc45b9acc1fc4aab5697e26a3f2ca57a763135c4f3\": container with ID starting with 891f85911a5b5604b88db6cc45b9acc1fc4aab5697e26a3f2ca57a763135c4f3 not found: ID does not exist" Nov 03 19:40:40.691594 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.691565 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-fn7j5-kgmpq"] Nov 03 19:40:40.697863 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.697841 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-fn7j5-kgmpq"] Nov 03 19:40:40.758030 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:40.757968 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-6x92p-7c5mm"] Nov 03 19:40:40.761080 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:40:40.761051 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2055fe6f_33ae_45f0_a7b0_7ccec390e4e5.slice/crio-79050eddb48e3723dfbeed3bb29477f3b62e16a459e97072faa132b7d3194998 WatchSource:0}: Error finding container 79050eddb48e3723dfbeed3bb29477f3b62e16a459e97072faa132b7d3194998: Status 404 returned error can't find the container with id 79050eddb48e3723dfbeed3bb29477f3b62e16a459e97072faa132b7d3194998 Nov 03 19:40:41.616928 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.616889 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-58c5cc9f56-qwcsc" podUID="1739913f-6ecd-4a90-80c6-ac0c7c36d224" containerName="console" containerID="cri-o://0d38791f1d88d628b4d854db5ea2458362b90f7c6369d8996336597e236ef9e3" gracePeriod=15 Nov 03 19:40:41.668217 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.668181 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6x92p-7c5mm" event={"ID":"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5","Type":"ContainerStarted","Data":"0c7dc570f6f2ebd00fee56af40af97b2b25491dc4259990628f0b85602a19cf0"} Nov 03 19:40:41.668605 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.668230 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6x92p-7c5mm" event={"ID":"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5","Type":"ContainerStarted","Data":"79050eddb48e3723dfbeed3bb29477f3b62e16a459e97072faa132b7d3194998"} Nov 03 19:40:41.685652 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.685604 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-6x92p-7c5mm" podStartSLOduration=1.685591465 podStartE2EDuration="1.685591465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:41.683939982 +0000 UTC m=+71.646318422" watchObservedRunningTime="2026-11-03 19:40:41.685591465 +0000 UTC m=+71.647969904" Nov 03 19:40:41.894895 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.894865 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:40:41.918122 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.918091 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-service-ca\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-service-ca\") pod \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " Nov 03 19:40:41.918303 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.918151 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-config\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-config\") pod \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " Nov 03 19:40:41.918303 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.918186 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-oauth-config\") pod \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " Nov 03 19:40:41.918303 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.918231 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-trusted-ca-bundle\") pod \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " Nov 03 19:40:41.918469 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.918383 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-oauth-serving-cert\") pod \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " Nov 03 19:40:41.918469 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.918451 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-serving-cert\") pod \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " Nov 03 19:40:41.918593 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.918500 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1739913f-6ecd-4a90-80c6-ac0c7c36d224-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1739913f-6ecd-4a90-80c6-ac0c7c36d224-tmp\") pod \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " Nov 03 19:40:41.918593 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.918552 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1739913f-6ecd-4a90-80c6-ac0c7c36d224-kube-api-access-n4d4p\" (UniqueName: \"kubernetes.io/projected/1739913f-6ecd-4a90-80c6-ac0c7c36d224-kube-api-access-n4d4p\") pod \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\" (UID: \"1739913f-6ecd-4a90-80c6-ac0c7c36d224\") " Nov 03 19:40:41.918593 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.918559 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-service-ca" pod "1739913f-6ecd-4a90-80c6-ac0c7c36d224" (UID: "1739913f-6ecd-4a90-80c6-ac0c7c36d224"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:41.918771 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.918626 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-config" pod "1739913f-6ecd-4a90-80c6-ac0c7c36d224" (UID: "1739913f-6ecd-4a90-80c6-ac0c7c36d224"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:41.918771 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.918734 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-trusted-ca-bundle" pod "1739913f-6ecd-4a90-80c6-ac0c7c36d224" (UID: "1739913f-6ecd-4a90-80c6-ac0c7c36d224"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:41.918984 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.918962 5111 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-config\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:41.919052 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.918992 5111 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-trusted-ca-bundle\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:41.919052 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.919009 5111 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-service-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:41.919052 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.919004 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1739913f-6ecd-4a90-80c6-ac0c7c36d224-tmp" pod "1739913f-6ecd-4a90-80c6-ac0c7c36d224" (UID: "1739913f-6ecd-4a90-80c6-ac0c7c36d224"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:40:41.919221 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.919197 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-oauth-serving-cert" pod "1739913f-6ecd-4a90-80c6-ac0c7c36d224" (UID: "1739913f-6ecd-4a90-80c6-ac0c7c36d224"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:40:41.920871 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.920840 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-oauth-config" pod "1739913f-6ecd-4a90-80c6-ac0c7c36d224" (UID: "1739913f-6ecd-4a90-80c6-ac0c7c36d224"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:41.921084 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.921069 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1739913f-6ecd-4a90-80c6-ac0c7c36d224-kube-api-access-n4d4p" pod "1739913f-6ecd-4a90-80c6-ac0c7c36d224" (UID: "1739913f-6ecd-4a90-80c6-ac0c7c36d224"). InnerVolumeSpecName "kube-api-access-n4d4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:40:41.921652 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:41.921628 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-serving-cert" pod "1739913f-6ecd-4a90-80c6-ac0c7c36d224" (UID: "1739913f-6ecd-4a90-80c6-ac0c7c36d224"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:40:42.019512 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:42.019478 5111 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-serving-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:42.019512 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:42.019510 5111 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1739913f-6ecd-4a90-80c6-ac0c7c36d224-tmp\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:42.019512 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:42.019520 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4d4p\" (UniqueName: \"kubernetes.io/projected/1739913f-6ecd-4a90-80c6-ac0c7c36d224-kube-api-access-n4d4p\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:42.019766 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:42.019555 5111 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1739913f-6ecd-4a90-80c6-ac0c7c36d224-console-oauth-config\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:42.019766 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:42.019567 5111 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1739913f-6ecd-4a90-80c6-ac0c7c36d224-oauth-serving-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:40:42.180203 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:42.180168 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467" path="/var/lib/kubelet/pods/c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467/volumes" Nov 03 19:40:42.674012 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:42.673978 5111 generic.go:356] "Generic (PLEG): container finished" podID="1739913f-6ecd-4a90-80c6-ac0c7c36d224" containerID="0d38791f1d88d628b4d854db5ea2458362b90f7c6369d8996336597e236ef9e3" exitCode=0 Nov 03 19:40:42.674522 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:42.674029 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c5cc9f56-qwcsc" Nov 03 19:40:42.674522 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:42.674070 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c5cc9f56-qwcsc" event={"ID":"1739913f-6ecd-4a90-80c6-ac0c7c36d224","Type":"ContainerDied","Data":"0d38791f1d88d628b4d854db5ea2458362b90f7c6369d8996336597e236ef9e3"} Nov 03 19:40:42.674522 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:42.674132 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c5cc9f56-qwcsc" event={"ID":"1739913f-6ecd-4a90-80c6-ac0c7c36d224","Type":"ContainerDied","Data":"14717762bb24715332126bd006de1121fe8ee89e5d5421ba262afb0f916db79c"} Nov 03 19:40:42.674522 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:42.674157 5111 scope.go:122] "RemoveContainer" containerID="0d38791f1d88d628b4d854db5ea2458362b90f7c6369d8996336597e236ef9e3" Nov 03 19:40:42.683009 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:42.682987 5111 scope.go:122] "RemoveContainer" containerID="0d38791f1d88d628b4d854db5ea2458362b90f7c6369d8996336597e236ef9e3" Nov 03 19:40:42.683296 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:40:42.683274 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d38791f1d88d628b4d854db5ea2458362b90f7c6369d8996336597e236ef9e3\": container with ID starting with 0d38791f1d88d628b4d854db5ea2458362b90f7c6369d8996336597e236ef9e3 not found: ID does not exist" containerID="0d38791f1d88d628b4d854db5ea2458362b90f7c6369d8996336597e236ef9e3" Nov 03 19:40:42.683349 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:42.683305 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d38791f1d88d628b4d854db5ea2458362b90f7c6369d8996336597e236ef9e3"} err="failed to get container status \"0d38791f1d88d628b4d854db5ea2458362b90f7c6369d8996336597e236ef9e3\": rpc error: code = NotFound desc = could not find container \"0d38791f1d88d628b4d854db5ea2458362b90f7c6369d8996336597e236ef9e3\": container with ID starting with 0d38791f1d88d628b4d854db5ea2458362b90f7c6369d8996336597e236ef9e3 not found: ID does not exist" Nov 03 19:40:42.697042 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:42.697016 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58c5cc9f56-qwcsc"] Nov 03 19:40:42.700704 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:42.700675 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58c5cc9f56-qwcsc"] Nov 03 19:40:44.180658 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:44.180626 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1739913f-6ecd-4a90-80c6-ac0c7c36d224" path="/var/lib/kubelet/pods/1739913f-6ecd-4a90-80c6-ac0c7c36d224/volumes" Nov 03 19:40:47.740740 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.740707 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-66dc498674-znblj"] Nov 03 19:40:47.741296 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.741161 5111 cpu_manager.go:414] "containerMap: removing container" podUID="1739913f-6ecd-4a90-80c6-ac0c7c36d224" containerName="console" Nov 03 19:40:47.741296 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.741181 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1739913f-6ecd-4a90-80c6-ac0c7c36d224" containerName="console" Nov 03 19:40:47.741296 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.741193 5111 cpu_manager.go:414] "containerMap: removing container" podUID="c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467" containerName="insights-gathering" Nov 03 19:40:47.741296 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.741202 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467" containerName="insights-gathering" Nov 03 19:40:47.741430 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.741310 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="1739913f-6ecd-4a90-80c6-ac0c7c36d224" containerName="console" Nov 03 19:40:47.741430 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.741322 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e1e0d8-a4ca-41be-a0cf-c3d2da4d7467" containerName="insights-gathering" Nov 03 19:40:47.768894 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.768858 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66dc498674-znblj"] Nov 03 19:40:47.769086 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.768915 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.871652 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.871606 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-tmp\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.871652 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.871650 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g49h5\" (UniqueName: \"kubernetes.io/projected/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-kube-api-access-g49h5\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.871892 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.871749 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-serving-cert\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.871892 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.871805 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-config\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.871892 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.871869 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-service-ca\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.872047 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.871892 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-oauth-serving-cert\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.872047 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.871932 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-oauth-config\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.872047 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.871989 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-trusted-ca-bundle\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.973248 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.973209 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-tmp\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.973248 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.973246 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g49h5\" (UniqueName: \"kubernetes.io/projected/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-kube-api-access-g49h5\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.973509 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.973273 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-serving-cert\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.973509 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.973304 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-config\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.973509 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.973420 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-service-ca\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.973509 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.973472 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-oauth-serving-cert\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.973509 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.973502 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-oauth-config\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.973816 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.973569 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-trusted-ca-bundle\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.973816 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.973751 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-tmp\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.974175 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.974152 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-config\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.974259 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.974159 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-service-ca\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.974318 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.974269 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-oauth-serving-cert\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.974391 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.974370 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-trusted-ca-bundle\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.976093 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.976074 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-serving-cert\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.976320 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.976286 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-oauth-config\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:47.981396 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:47.981371 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g49h5\" (UniqueName: \"kubernetes.io/projected/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-kube-api-access-g49h5\") pod \"console-66dc498674-znblj\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:48.099194 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:48.099109 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:48.405706 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:48.405528 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66dc498674-znblj"] Nov 03 19:40:48.411666 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:40:48.411630 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7e03de_094b_461c_ade9_f3bbe1a5a7f5.slice/crio-c60ed598f06c3d796fd7bd1ed45589d7658d495609d0099da555301bc714d106 WatchSource:0}: Error finding container c60ed598f06c3d796fd7bd1ed45589d7658d495609d0099da555301bc714d106: Status 404 returned error can't find the container with id c60ed598f06c3d796fd7bd1ed45589d7658d495609d0099da555301bc714d106 Nov 03 19:40:48.703244 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:48.703148 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66dc498674-znblj" event={"ID":"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5","Type":"ContainerStarted","Data":"3978c57d866a5dede4d7e8051aff55ac425aa85eddc9d786e7bf2fc7c75b941a"} Nov 03 19:40:48.703244 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:48.703212 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66dc498674-znblj" event={"ID":"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5","Type":"ContainerStarted","Data":"c60ed598f06c3d796fd7bd1ed45589d7658d495609d0099da555301bc714d106"} Nov 03 19:40:48.721819 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:48.721767 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-66dc498674-znblj" podStartSLOduration=1.721753997 podStartE2EDuration="1.721753997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:48.719766905 +0000 UTC m=+78.682145344" watchObservedRunningTime="2026-11-03 19:40:48.721753997 +0000 UTC m=+78.684132436" Nov 03 19:40:58.100191 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:58.100148 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:58.100719 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:58.100224 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:58.104954 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:58.104933 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:58.748164 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:58.748127 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66dc498674-znblj" Nov 03 19:40:58.791732 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:40:58.791699 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f9cc58d48-kmlkw"] Nov 03 19:41:05.054680 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:41:05.054627 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:41:08.504800 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.504760 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-69f57dcd5b-8j7f7"] Nov 03 19:41:08.508266 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.508244 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.548985 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.548956 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69f57dcd5b-8j7f7"] Nov 03 19:41:08.601919 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.601882 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c53940d4-bf84-4e09-a5ee-63a02d237c09-tmp\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.601919 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.601922 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glgzd\" (UniqueName: \"kubernetes.io/projected/c53940d4-bf84-4e09-a5ee-63a02d237c09-kube-api-access-glgzd\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.602141 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.602004 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c53940d4-bf84-4e09-a5ee-63a02d237c09-console-serving-cert\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.602141 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.602039 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c53940d4-bf84-4e09-a5ee-63a02d237c09-console-config\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.602141 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.602072 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c53940d4-bf84-4e09-a5ee-63a02d237c09-service-ca\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.602324 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.602138 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c53940d4-bf84-4e09-a5ee-63a02d237c09-trusted-ca-bundle\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.602324 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.602269 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c53940d4-bf84-4e09-a5ee-63a02d237c09-oauth-serving-cert\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.602416 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.602323 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c53940d4-bf84-4e09-a5ee-63a02d237c09-console-oauth-config\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.703021 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.702983 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c53940d4-bf84-4e09-a5ee-63a02d237c09-tmp\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.703214 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.703027 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glgzd\" (UniqueName: \"kubernetes.io/projected/c53940d4-bf84-4e09-a5ee-63a02d237c09-kube-api-access-glgzd\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.703214 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.703072 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c53940d4-bf84-4e09-a5ee-63a02d237c09-console-serving-cert\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.703214 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.703097 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c53940d4-bf84-4e09-a5ee-63a02d237c09-console-config\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.703214 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.703134 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c53940d4-bf84-4e09-a5ee-63a02d237c09-service-ca\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.703214 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.703155 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c53940d4-bf84-4e09-a5ee-63a02d237c09-trusted-ca-bundle\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.703214 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.703188 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c53940d4-bf84-4e09-a5ee-63a02d237c09-oauth-serving-cert\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.703520 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.703226 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c53940d4-bf84-4e09-a5ee-63a02d237c09-console-oauth-config\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.703520 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.703424 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c53940d4-bf84-4e09-a5ee-63a02d237c09-tmp\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.703954 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.703927 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c53940d4-bf84-4e09-a5ee-63a02d237c09-console-config\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.704075 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.703934 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c53940d4-bf84-4e09-a5ee-63a02d237c09-service-ca\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.704139 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.704090 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c53940d4-bf84-4e09-a5ee-63a02d237c09-oauth-serving-cert\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.704204 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.704166 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c53940d4-bf84-4e09-a5ee-63a02d237c09-trusted-ca-bundle\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.705706 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.705680 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c53940d4-bf84-4e09-a5ee-63a02d237c09-console-serving-cert\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.705803 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.705708 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c53940d4-bf84-4e09-a5ee-63a02d237c09-console-oauth-config\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.714672 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.714652 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glgzd\" (UniqueName: \"kubernetes.io/projected/c53940d4-bf84-4e09-a5ee-63a02d237c09-kube-api-access-glgzd\") pod \"console-69f57dcd5b-8j7f7\" (UID: \"c53940d4-bf84-4e09-a5ee-63a02d237c09\") " pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:08.819868 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:08.819792 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:09.094135 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:09.094109 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69f57dcd5b-8j7f7"] Nov 03 19:41:09.096817 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:41:09.096790 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53940d4_bf84_4e09_a5ee_63a02d237c09.slice/crio-a23f43ea15a85fdc16807a1cc1ea59c4fb0181caca73f208b75caa0d59b17017 WatchSource:0}: Error finding container a23f43ea15a85fdc16807a1cc1ea59c4fb0181caca73f208b75caa0d59b17017: Status 404 returned error can't find the container with id a23f43ea15a85fdc16807a1cc1ea59c4fb0181caca73f208b75caa0d59b17017 Nov 03 19:41:09.793491 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:09.793454 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f57dcd5b-8j7f7" event={"ID":"c53940d4-bf84-4e09-a5ee-63a02d237c09","Type":"ContainerStarted","Data":"e83269772335c0770441fe2bf190fe0ce96fe0c5d273de49fdc2d34313ce231a"} Nov 03 19:41:09.793910 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:09.793504 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f57dcd5b-8j7f7" event={"ID":"c53940d4-bf84-4e09-a5ee-63a02d237c09","Type":"ContainerStarted","Data":"a23f43ea15a85fdc16807a1cc1ea59c4fb0181caca73f208b75caa0d59b17017"} Nov 03 19:41:09.837590 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:09.837517 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-69f57dcd5b-8j7f7" podStartSLOduration=1.837502691 podStartE2EDuration="1.837502691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:09.837306177 +0000 UTC m=+99.799684616" watchObservedRunningTime="2026-11-03 19:41:09.837502691 +0000 UTC m=+99.799881216" Nov 03 19:41:18.820459 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:18.820407 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:18.820997 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:18.820483 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:18.825258 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:18.825232 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:18.835859 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:18.835828 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69f57dcd5b-8j7f7" Nov 03 19:41:18.894174 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:18.894138 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66dc498674-znblj"] Nov 03 19:41:23.816267 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:23.816225 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-f9cc58d48-kmlkw" podUID="e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b" containerName="console" containerID="cri-o://9dfe8d059be6cc08cbdf23adf9fe700ed51ec1e246179474b4c163e401fc056e" gracePeriod=15 Nov 03 19:41:24.072454 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.072389 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:41:24.139288 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.139252 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-service-ca\") pod \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " Nov 03 19:41:24.139472 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.139299 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-oauth-serving-cert\") pod \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " Nov 03 19:41:24.139472 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.139351 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-kube-api-access-tgxsw\" (UniqueName: \"kubernetes.io/projected/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-kube-api-access-tgxsw\") pod \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " Nov 03 19:41:24.139472 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.139372 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-trusted-ca-bundle\") pod \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " Nov 03 19:41:24.139472 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.139399 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-serving-cert\") pod \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " Nov 03 19:41:24.139472 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.139419 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-tmp\") pod \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " Nov 03 19:41:24.139472 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.139461 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-oauth-config\") pod \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " Nov 03 19:41:24.139896 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.139480 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-config\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-config\") pod \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\" (UID: \"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b\") " Nov 03 19:41:24.139896 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.139747 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-service-ca" pod "e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b" (UID: "e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:24.140005 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.139900 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-trusted-ca-bundle" pod "e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b" (UID: "e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:24.140005 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.139911 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-oauth-serving-cert" pod "e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b" (UID: "e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:24.140115 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.140001 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-config" pod "e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b" (UID: "e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:24.140115 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.140087 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-tmp" pod "e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b" (UID: "e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:41:24.141811 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.141781 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-oauth-config" pod "e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b" (UID: "e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:41:24.141902 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.141815 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-serving-cert" pod "e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b" (UID: "e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:41:24.141902 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.141819 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-kube-api-access-tgxsw" pod "e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b" (UID: "e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b"). InnerVolumeSpecName "kube-api-access-tgxsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:41:24.240246 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.240213 5111 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-service-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:24.240246 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.240246 5111 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-oauth-serving-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:24.240471 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.240259 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgxsw\" (UniqueName: \"kubernetes.io/projected/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-kube-api-access-tgxsw\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:24.240471 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.240273 5111 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-trusted-ca-bundle\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:24.240471 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.240285 5111 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-serving-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:24.240471 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.240298 5111 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-tmp\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:24.240471 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.240310 5111 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-oauth-config\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:24.240471 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.240322 5111 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b-console-config\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:24.858150 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.858113 5111 generic.go:356] "Generic (PLEG): container finished" podID="e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b" containerID="9dfe8d059be6cc08cbdf23adf9fe700ed51ec1e246179474b4c163e401fc056e" exitCode=0 Nov 03 19:41:24.858580 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.858181 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9cc58d48-kmlkw" Nov 03 19:41:24.858580 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.858200 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9cc58d48-kmlkw" event={"ID":"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b","Type":"ContainerDied","Data":"9dfe8d059be6cc08cbdf23adf9fe700ed51ec1e246179474b4c163e401fc056e"} Nov 03 19:41:24.858580 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.858286 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9cc58d48-kmlkw" event={"ID":"e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b","Type":"ContainerDied","Data":"2f98afaf5addf810ee7e05800655ed3c5bd96603e22d098e5c6420674f45e099"} Nov 03 19:41:24.858580 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.858314 5111 scope.go:122] "RemoveContainer" containerID="9dfe8d059be6cc08cbdf23adf9fe700ed51ec1e246179474b4c163e401fc056e" Nov 03 19:41:24.866896 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.866876 5111 scope.go:122] "RemoveContainer" containerID="9dfe8d059be6cc08cbdf23adf9fe700ed51ec1e246179474b4c163e401fc056e" Nov 03 19:41:24.867192 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:41:24.867172 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dfe8d059be6cc08cbdf23adf9fe700ed51ec1e246179474b4c163e401fc056e\": container with ID starting with 9dfe8d059be6cc08cbdf23adf9fe700ed51ec1e246179474b4c163e401fc056e not found: ID does not exist" containerID="9dfe8d059be6cc08cbdf23adf9fe700ed51ec1e246179474b4c163e401fc056e" Nov 03 19:41:24.867246 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.867202 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dfe8d059be6cc08cbdf23adf9fe700ed51ec1e246179474b4c163e401fc056e"} err="failed to get container status \"9dfe8d059be6cc08cbdf23adf9fe700ed51ec1e246179474b4c163e401fc056e\": rpc error: code = NotFound desc = could not find container \"9dfe8d059be6cc08cbdf23adf9fe700ed51ec1e246179474b4c163e401fc056e\": container with ID starting with 9dfe8d059be6cc08cbdf23adf9fe700ed51ec1e246179474b4c163e401fc056e not found: ID does not exist" Nov 03 19:41:24.875199 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.875172 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f9cc58d48-kmlkw"] Nov 03 19:41:24.878649 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:24.878626 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f9cc58d48-kmlkw"] Nov 03 19:41:26.180326 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:26.180290 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b" path="/var/lib/kubelet/pods/e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b/volumes" Nov 03 19:41:35.054905 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:41:35.054862 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:41:43.916275 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:43.916160 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-66dc498674-znblj" podUID="5a7e03de-094b-461c-ade9-f3bbe1a5a7f5" containerName="console" containerID="cri-o://3978c57d866a5dede4d7e8051aff55ac425aa85eddc9d786e7bf2fc7c75b941a" gracePeriod=15 Nov 03 19:41:44.164333 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.164311 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66dc498674-znblj" Nov 03 19:41:44.208432 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.208368 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-oauth-config\") pod \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " Nov 03 19:41:44.208432 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.208400 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-oauth-serving-cert\") pod \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " Nov 03 19:41:44.208661 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.208480 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-config\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-config\") pod \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " Nov 03 19:41:44.208661 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.208504 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-service-ca\") pod \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " Nov 03 19:41:44.208661 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.208560 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-serving-cert\") pod \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " Nov 03 19:41:44.208661 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.208610 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-trusted-ca-bundle\") pod \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " Nov 03 19:41:44.208661 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.208645 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-kube-api-access-g49h5\" (UniqueName: \"kubernetes.io/projected/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-kube-api-access-g49h5\") pod \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " Nov 03 19:41:44.208949 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.208685 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-tmp\") pod \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\" (UID: \"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5\") " Nov 03 19:41:44.208949 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.208899 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-oauth-serving-cert" pod "5a7e03de-094b-461c-ade9-f3bbe1a5a7f5" (UID: "5a7e03de-094b-461c-ade9-f3bbe1a5a7f5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:44.208949 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.208915 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-config" pod "5a7e03de-094b-461c-ade9-f3bbe1a5a7f5" (UID: "5a7e03de-094b-461c-ade9-f3bbe1a5a7f5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:44.209205 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.209174 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-service-ca" pod "5a7e03de-094b-461c-ade9-f3bbe1a5a7f5" (UID: "5a7e03de-094b-461c-ade9-f3bbe1a5a7f5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:44.209336 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.209283 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-trusted-ca-bundle" pod "5a7e03de-094b-461c-ade9-f3bbe1a5a7f5" (UID: "5a7e03de-094b-461c-ade9-f3bbe1a5a7f5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:44.209405 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.209370 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-tmp" pod "5a7e03de-094b-461c-ade9-f3bbe1a5a7f5" (UID: "5a7e03de-094b-461c-ade9-f3bbe1a5a7f5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:41:44.211024 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.211000 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-oauth-config" pod "5a7e03de-094b-461c-ade9-f3bbe1a5a7f5" (UID: "5a7e03de-094b-461c-ade9-f3bbe1a5a7f5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:41:44.211092 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.211020 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-serving-cert" pod "5a7e03de-094b-461c-ade9-f3bbe1a5a7f5" (UID: "5a7e03de-094b-461c-ade9-f3bbe1a5a7f5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:41:44.211147 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.211124 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-kube-api-access-g49h5" pod "5a7e03de-094b-461c-ade9-f3bbe1a5a7f5" (UID: "5a7e03de-094b-461c-ade9-f3bbe1a5a7f5"). InnerVolumeSpecName "kube-api-access-g49h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:41:44.309398 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.309369 5111 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-tmp\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:44.309398 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.309397 5111 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-oauth-config\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:44.309589 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.309407 5111 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-oauth-serving-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:44.309589 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.309416 5111 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-config\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:44.309589 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.309425 5111 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-service-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:44.309589 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.309432 5111 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-console-serving-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:44.309589 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.309441 5111 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-trusted-ca-bundle\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:44.309589 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.309449 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g49h5\" (UniqueName: \"kubernetes.io/projected/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5-kube-api-access-g49h5\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:44.953193 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.953159 5111 generic.go:356] "Generic (PLEG): container finished" podID="5a7e03de-094b-461c-ade9-f3bbe1a5a7f5" containerID="3978c57d866a5dede4d7e8051aff55ac425aa85eddc9d786e7bf2fc7c75b941a" exitCode=0 Nov 03 19:41:44.953623 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.953207 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66dc498674-znblj" Nov 03 19:41:44.953623 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.953212 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66dc498674-znblj" event={"ID":"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5","Type":"ContainerDied","Data":"3978c57d866a5dede4d7e8051aff55ac425aa85eddc9d786e7bf2fc7c75b941a"} Nov 03 19:41:44.953623 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.953266 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66dc498674-znblj" event={"ID":"5a7e03de-094b-461c-ade9-f3bbe1a5a7f5","Type":"ContainerDied","Data":"c60ed598f06c3d796fd7bd1ed45589d7658d495609d0099da555301bc714d106"} Nov 03 19:41:44.953623 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.953286 5111 scope.go:122] "RemoveContainer" containerID="3978c57d866a5dede4d7e8051aff55ac425aa85eddc9d786e7bf2fc7c75b941a" Nov 03 19:41:44.962939 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.962920 5111 scope.go:122] "RemoveContainer" containerID="3978c57d866a5dede4d7e8051aff55ac425aa85eddc9d786e7bf2fc7c75b941a" Nov 03 19:41:44.963199 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:41:44.963182 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3978c57d866a5dede4d7e8051aff55ac425aa85eddc9d786e7bf2fc7c75b941a\": container with ID starting with 3978c57d866a5dede4d7e8051aff55ac425aa85eddc9d786e7bf2fc7c75b941a not found: ID does not exist" containerID="3978c57d866a5dede4d7e8051aff55ac425aa85eddc9d786e7bf2fc7c75b941a" Nov 03 19:41:44.963243 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.963207 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3978c57d866a5dede4d7e8051aff55ac425aa85eddc9d786e7bf2fc7c75b941a"} err="failed to get container status \"3978c57d866a5dede4d7e8051aff55ac425aa85eddc9d786e7bf2fc7c75b941a\": rpc error: code = NotFound desc = could not find container \"3978c57d866a5dede4d7e8051aff55ac425aa85eddc9d786e7bf2fc7c75b941a\": container with ID starting with 3978c57d866a5dede4d7e8051aff55ac425aa85eddc9d786e7bf2fc7c75b941a not found: ID does not exist" Nov 03 19:41:44.975003 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.974982 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66dc498674-znblj"] Nov 03 19:41:44.979029 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:44.979008 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66dc498674-znblj"] Nov 03 19:41:46.183026 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:41:46.182990 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a7e03de-094b-461c-ade9-f3bbe1a5a7f5" path="/var/lib/kubelet/pods/5a7e03de-094b-461c-ade9-f3bbe1a5a7f5/volumes" Nov 03 19:42:05.039632 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:05.039593 5111 generic.go:356] "Generic (PLEG): container finished" podID="2055fe6f-33ae-45f0-a7b0-7ccec390e4e5" containerID="0c7dc570f6f2ebd00fee56af40af97b2b25491dc4259990628f0b85602a19cf0" exitCode=0 Nov 03 19:42:05.040254 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:05.039663 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6x92p-7c5mm" event={"ID":"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5","Type":"ContainerDied","Data":"0c7dc570f6f2ebd00fee56af40af97b2b25491dc4259990628f0b85602a19cf0"} Nov 03 19:42:05.056149 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:42:05.056117 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:42:06.181417 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:06.181392 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6x92p-7c5mm" Nov 03 19:42:06.282607 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:06.282577 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-archives-path\") pod \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\" (UID: \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\") " Nov 03 19:42:06.282845 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:06.282639 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-trusted-ca-bundle\") pod \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\" (UID: \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\") " Nov 03 19:42:06.282845 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:06.282743 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-service-ca-bundle\") pod \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\" (UID: \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\") " Nov 03 19:42:06.282845 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:06.282786 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-kube-api-access-v4fc2\" (UniqueName: \"kubernetes.io/projected/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-kube-api-access-v4fc2\") pod \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\" (UID: \"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5\") " Nov 03 19:42:06.283008 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:06.282893 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-archives-path" pod "2055fe6f-33ae-45f0-a7b0-7ccec390e4e5" (UID: "2055fe6f-33ae-45f0-a7b0-7ccec390e4e5"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:42:06.283085 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:06.283058 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-trusted-ca-bundle" pod "2055fe6f-33ae-45f0-a7b0-7ccec390e4e5" (UID: "2055fe6f-33ae-45f0-a7b0-7ccec390e4e5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:06.283171 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:06.283097 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-service-ca-bundle" pod "2055fe6f-33ae-45f0-a7b0-7ccec390e4e5" (UID: "2055fe6f-33ae-45f0-a7b0-7ccec390e4e5"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:06.283171 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:06.283140 5111 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-trusted-ca-bundle\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:06.283171 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:06.283164 5111 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-archives-path\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:06.284917 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:06.284898 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-kube-api-access-v4fc2" pod "2055fe6f-33ae-45f0-a7b0-7ccec390e4e5" (UID: "2055fe6f-33ae-45f0-a7b0-7ccec390e4e5"). InnerVolumeSpecName "kube-api-access-v4fc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:06.383457 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:06.383429 5111 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-service-ca-bundle\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:06.383588 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:06.383458 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4fc2\" (UniqueName: \"kubernetes.io/projected/2055fe6f-33ae-45f0-a7b0-7ccec390e4e5-kube-api-access-v4fc2\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:07.048803 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:07.048769 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6x92p-7c5mm" Nov 03 19:42:07.048803 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:07.048796 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6x92p-7c5mm" event={"ID":"2055fe6f-33ae-45f0-a7b0-7ccec390e4e5","Type":"ContainerDied","Data":"79050eddb48e3723dfbeed3bb29477f3b62e16a459e97072faa132b7d3194998"} Nov 03 19:42:07.049023 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:42:07.048841 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79050eddb48e3723dfbeed3bb29477f3b62e16a459e97072faa132b7d3194998" Nov 03 19:42:35.054891 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:42:35.054852 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:43:05.054777 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:43:05.054725 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:43:35.055214 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:43:35.055175 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:44:05.055444 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:44:05.055404 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:44:30.195747 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:44:30.195715 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 19:44:30.197394 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:44:30.197373 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 19:44:30.200639 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:44:30.200618 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 19:44:30.202573 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:44:30.202554 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 19:44:35.054910 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:44:35.054876 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:45:05.054967 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:45:05.054928 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:45:14.200088 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.200049 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c"] Nov 03 19:45:14.200803 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.200547 5111 cpu_manager.go:414] "containerMap: removing container" podUID="2055fe6f-33ae-45f0-a7b0-7ccec390e4e5" containerName="insights-gathering" Nov 03 19:45:14.200803 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.200567 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2055fe6f-33ae-45f0-a7b0-7ccec390e4e5" containerName="insights-gathering" Nov 03 19:45:14.200803 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.200585 5111 cpu_manager.go:414] "containerMap: removing container" podUID="5a7e03de-094b-461c-ade9-f3bbe1a5a7f5" containerName="console" Nov 03 19:45:14.200803 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.200593 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a7e03de-094b-461c-ade9-f3bbe1a5a7f5" containerName="console" Nov 03 19:45:14.200803 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.200614 5111 cpu_manager.go:414] "containerMap: removing container" podUID="e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b" containerName="console" Nov 03 19:45:14.200803 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.200623 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b" containerName="console" Nov 03 19:45:14.200803 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.200734 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="2055fe6f-33ae-45f0-a7b0-7ccec390e4e5" containerName="insights-gathering" Nov 03 19:45:14.200803 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.200748 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a7e03de-094b-461c-ade9-f3bbe1a5a7f5" containerName="console" Nov 03 19:45:14.200803 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.200759 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2cb9475-747e-460b-b7fc-8c4bd6ff0b5b" containerName="console" Nov 03 19:45:14.204101 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.204077 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.215703 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.215678 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 19:45:14.216772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.216746 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dkm76\"" Nov 03 19:45:14.218452 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.218425 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c"] Nov 03 19:45:14.282764 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.282729 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f"] Nov 03 19:45:14.286120 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.286096 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.298266 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.298232 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-m4zdm\"" Nov 03 19:45:14.305928 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.305896 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f"] Nov 03 19:45:14.306255 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.306225 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.306398 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.306275 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.306398 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.306381 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f667fce-363f-476e-8601-afd78a7f3121-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.306502 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.306439 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.306502 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.306467 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f667fce-363f-476e-8601-afd78a7f3121-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.306502 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.306490 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.306679 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.306519 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-895wn\" (UniqueName: \"kubernetes.io/projected/0f667fce-363f-476e-8601-afd78a7f3121-kube-api-access-895wn\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.306679 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.306565 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.306679 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.306600 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f667fce-363f-476e-8601-afd78a7f3121-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.407248 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407209 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.407442 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407254 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.407442 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407292 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ed7c10-e620-4856-b654-866ee1fc916f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.407442 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407315 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ed7c10-e620-4856-b654-866ee1fc916f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.407442 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407341 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ed7c10-e620-4856-b654-866ee1fc916f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.407442 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407364 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwrdx\" (UniqueName: \"kubernetes.io/projected/65ed7c10-e620-4856-b654-866ee1fc916f-kube-api-access-kwrdx\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.407697 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407475 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f667fce-363f-476e-8601-afd78a7f3121-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.407697 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407515 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ed7c10-e620-4856-b654-866ee1fc916f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.407697 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407560 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ed7c10-e620-4856-b654-866ee1fc916f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.407697 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407629 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ed7c10-e620-4856-b654-866ee1fc916f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.407697 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407661 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ed7c10-e620-4856-b654-866ee1fc916f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.407920 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407701 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.407920 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407707 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ed7c10-e620-4856-b654-866ee1fc916f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.407920 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407710 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.407920 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407821 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.407920 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407854 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f667fce-363f-476e-8601-afd78a7f3121-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.407920 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407883 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.407920 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407917 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-895wn\" (UniqueName: \"kubernetes.io/projected/0f667fce-363f-476e-8601-afd78a7f3121-kube-api-access-895wn\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.408265 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407947 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.408265 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.407975 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f667fce-363f-476e-8601-afd78a7f3121-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.408265 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.408113 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.408265 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.408220 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f667fce-363f-476e-8601-afd78a7f3121-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.408449 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.408314 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.410166 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.410141 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.410356 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.410338 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f667fce-363f-476e-8601-afd78a7f3121-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.415907 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.415884 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-895wn\" (UniqueName: \"kubernetes.io/projected/0f667fce-363f-476e-8601-afd78a7f3121-kube-api-access-895wn\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.415999 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.415884 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f667fce-363f-476e-8601-afd78a7f3121-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-p994c\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.508947 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.508854 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ed7c10-e620-4856-b654-866ee1fc916f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.508947 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.508900 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ed7c10-e620-4856-b654-866ee1fc916f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.508947 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.508932 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ed7c10-e620-4856-b654-866ee1fc916f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.509228 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.508957 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ed7c10-e620-4856-b654-866ee1fc916f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.509228 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.508994 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ed7c10-e620-4856-b654-866ee1fc916f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.509228 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.509091 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ed7c10-e620-4856-b654-866ee1fc916f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.509228 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.509113 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ed7c10-e620-4856-b654-866ee1fc916f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.509228 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.509140 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ed7c10-e620-4856-b654-866ee1fc916f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.509228 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.509167 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrdx\" (UniqueName: \"kubernetes.io/projected/65ed7c10-e620-4856-b654-866ee1fc916f-kube-api-access-kwrdx\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.509482 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.509289 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ed7c10-e620-4856-b654-866ee1fc916f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.509482 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.509360 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ed7c10-e620-4856-b654-866ee1fc916f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.509614 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.509485 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ed7c10-e620-4856-b654-866ee1fc916f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.509667 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.509615 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ed7c10-e620-4856-b654-866ee1fc916f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.509909 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.509887 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ed7c10-e620-4856-b654-866ee1fc916f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.511322 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.511296 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ed7c10-e620-4856-b654-866ee1fc916f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.511491 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.511476 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ed7c10-e620-4856-b654-866ee1fc916f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.519084 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.519059 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:14.525596 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.525574 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ed7c10-e620-4856-b654-866ee1fc916f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.528147 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.528126 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwrdx\" (UniqueName: \"kubernetes.io/projected/65ed7c10-e620-4856-b654-866ee1fc916f-kube-api-access-kwrdx\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f\" (UID: \"65ed7c10-e620-4856-b654-866ee1fc916f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.599884 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.599847 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:14.795784 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.795759 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c"] Nov 03 19:45:14.801640 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.801619 5111 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:45:14.876460 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:14.876433 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f"] Nov 03 19:45:14.878939 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:45:14.878910 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ed7c10_e620_4856_b654_866ee1fc916f.slice/crio-9d8a01a07afa4b1cf075bd14b1a9f6a3914e86c977383d7cad9ce337bb3013a2 WatchSource:0}: Error finding container 9d8a01a07afa4b1cf075bd14b1a9f6a3914e86c977383d7cad9ce337bb3013a2: Status 404 returned error can't find the container with id 9d8a01a07afa4b1cf075bd14b1a9f6a3914e86c977383d7cad9ce337bb3013a2 Nov 03 19:45:15.794636 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:15.794589 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" event={"ID":"0f667fce-363f-476e-8601-afd78a7f3121","Type":"ContainerStarted","Data":"d856bdbe3c6ef4c3164beb346b1a0a4b4c1f2d668e07b3df92609688cfef4b07"} Nov 03 19:45:15.795668 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:15.795635 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" event={"ID":"65ed7c10-e620-4856-b654-866ee1fc916f","Type":"ContainerStarted","Data":"9d8a01a07afa4b1cf075bd14b1a9f6a3914e86c977383d7cad9ce337bb3013a2"} Nov 03 19:45:18.567331 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:18.567297 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:45:18.567682 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:18.567367 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:45:18.567682 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:18.567401 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:45:18.816013 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:18.815922 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" event={"ID":"0f667fce-363f-476e-8601-afd78a7f3121","Type":"ContainerStarted","Data":"556de97197debf90d7ad5b1f2188cfa171a2e726d25c11fb54e706de9ecdb3b0"} Nov 03 19:45:18.837389 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:18.837327 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" podStartSLOduration=1.072591436 podStartE2EDuration="4.837311332s" totalImagesPullingTime="3.764719896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:18.836466247 +0000 UTC m=+348.798844686" watchObservedRunningTime="2026-11-03 19:45:18.837311332 +0000 UTC m=+348.799689774" Nov 03 19:45:18.927271 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:18.927237 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:45:18.927350 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:18.927312 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:45:18.927350 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:18.927340 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:45:19.520297 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:19.520257 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:19.525415 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:19.525386 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:19.821081 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:19.820988 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" event={"ID":"65ed7c10-e620-4856-b654-866ee1fc916f","Type":"ContainerStarted","Data":"3115a94c170b1d24b33f4cb6682b8f4ca710d8ee220f72c5ecf28dc6516fae5e"} Nov 03 19:45:19.821496 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:19.821263 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:19.822320 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:19.822302 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:45:19.839924 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:19.839874 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" podStartSLOduration=1.794081776 podStartE2EDuration="5.839860633s" totalImagesPullingTime="4.045778857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:19.838966502 +0000 UTC m=+349.801344965" watchObservedRunningTime="2026-11-03 19:45:19.839860633 +0000 UTC m=+349.802239071" Nov 03 19:45:20.600882 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:20.600840 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:20.605654 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:20.605624 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:20.825254 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:20.825214 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:20.826297 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:20.826276 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f" Nov 03 19:45:35.054797 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:45:35.054757 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:45:48.628036 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.628000 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6"] Nov 03 19:45:48.631735 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.631717 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.661667 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.661626 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6"] Nov 03 19:45:48.696917 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.696877 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 19:45:48.696917 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.696916 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 19:45:48.697182 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.697134 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 19:45:48.697182 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.697135 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 19:45:48.697182 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.697152 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 19:45:48.697338 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.697275 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-85znx\"" Nov 03 19:45:48.700863 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.700835 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0800588-b452-4a3d-ab60-e0a522174a33-manifests\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.701013 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.700870 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0800588-b452-4a3d-ab60-e0a522174a33-webhook-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.701013 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.700897 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0800588-b452-4a3d-ab60-e0a522174a33-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.701138 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.701107 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0800588-b452-4a3d-ab60-e0a522174a33-metrics-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.701199 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.701137 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0800588-b452-4a3d-ab60-e0a522174a33-tmp\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.701199 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.701170 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt7vj\" (UniqueName: \"kubernetes.io/projected/e0800588-b452-4a3d-ab60-e0a522174a33-kube-api-access-pt7vj\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.802300 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.802260 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0800588-b452-4a3d-ab60-e0a522174a33-manifests\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.802300 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.802297 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0800588-b452-4a3d-ab60-e0a522174a33-webhook-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.802688 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.802324 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0800588-b452-4a3d-ab60-e0a522174a33-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.802688 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.802380 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0800588-b452-4a3d-ab60-e0a522174a33-metrics-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.802688 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.802404 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0800588-b452-4a3d-ab60-e0a522174a33-tmp\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.802688 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.802433 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt7vj\" (UniqueName: \"kubernetes.io/projected/e0800588-b452-4a3d-ab60-e0a522174a33-kube-api-access-pt7vj\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.802910 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.802793 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0800588-b452-4a3d-ab60-e0a522174a33-manifests\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.804917 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.804892 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0800588-b452-4a3d-ab60-e0a522174a33-tmp\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.805043 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.805000 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0800588-b452-4a3d-ab60-e0a522174a33-webhook-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.805116 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.805075 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0800588-b452-4a3d-ab60-e0a522174a33-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.805209 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.805186 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0800588-b452-4a3d-ab60-e0a522174a33-metrics-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.814347 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.814319 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt7vj\" (UniqueName: \"kubernetes.io/projected/e0800588-b452-4a3d-ab60-e0a522174a33-kube-api-access-pt7vj\") pod \"opendatahub-operator-controller-manager-5749d45fbb-7cpx6\" (UID: \"e0800588-b452-4a3d-ab60-e0a522174a33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:48.947221 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:48.947126 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:49.244634 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:49.244563 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6"] Nov 03 19:45:49.248948 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:45:49.248920 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0800588_b452_4a3d_ab60_e0a522174a33.slice/crio-d01ed3e03bbb66da8461c8221875108e019e6505e2293c09f2e99f4b781cecf5 WatchSource:0}: Error finding container d01ed3e03bbb66da8461c8221875108e019e6505e2293c09f2e99f4b781cecf5: Status 404 returned error can't find the container with id d01ed3e03bbb66da8461c8221875108e019e6505e2293c09f2e99f4b781cecf5 Nov 03 19:45:49.952390 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:49.952349 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" event={"ID":"e0800588-b452-4a3d-ab60-e0a522174a33","Type":"ContainerStarted","Data":"d01ed3e03bbb66da8461c8221875108e019e6505e2293c09f2e99f4b781cecf5"} Nov 03 19:45:52.968178 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:52.968135 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" event={"ID":"e0800588-b452-4a3d-ab60-e0a522174a33","Type":"ContainerStarted","Data":"e7a6bf4116918cc716cc06084decb9588eeb3c6496798164013f404b1b669420"} Nov 03 19:45:53.973460 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:53.973422 5111 generic.go:356] "Generic (PLEG): container finished" podID="e0800588-b452-4a3d-ab60-e0a522174a33" containerID="e7a6bf4116918cc716cc06084decb9588eeb3c6496798164013f404b1b669420" exitCode=0 Nov 03 19:45:53.973897 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:53.973474 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" event={"ID":"e0800588-b452-4a3d-ab60-e0a522174a33","Type":"ContainerDied","Data":"e7a6bf4116918cc716cc06084decb9588eeb3c6496798164013f404b1b669420"} Nov 03 19:45:54.979747 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:54.979712 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" event={"ID":"e0800588-b452-4a3d-ab60-e0a522174a33","Type":"ContainerStarted","Data":"f83f8b0a1c57756aaafb98085d85fbf3e5f295467e8552995e18eb6c1e10bd71"} Nov 03 19:45:54.980179 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:54.979811 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:45:55.014720 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:45:55.014662 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" podStartSLOduration=2.59047419 podStartE2EDuration="7.014647815s" totalImagesPullingTime="3.931147424s" totalInitContainerRuntime="493.026201ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:55.012903541 +0000 UTC m=+384.975281980" watchObservedRunningTime="2026-11-03 19:45:55.014647815 +0000 UTC m=+384.977026245" Nov 03 19:46:05.054953 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:46:05.054914 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:46:05.987192 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:05.987157 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" Nov 03 19:46:13.324139 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:13.324057 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4hkl8"] Nov 03 19:46:13.330257 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:13.330237 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4hkl8" Nov 03 19:46:13.336025 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:13.335996 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4hkl8"] Nov 03 19:46:13.337302 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:13.337221 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 19:46:13.338408 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:13.338389 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 19:46:13.338408 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:13.338403 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5d7lw\"" Nov 03 19:46:13.426481 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:13.426450 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea879f58-8507-4415-bb98-1878031012e2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4hkl8\" (UID: \"ea879f58-8507-4415-bb98-1878031012e2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4hkl8" Nov 03 19:46:13.426661 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:13.426493 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d8m7\" (UniqueName: \"kubernetes.io/projected/ea879f58-8507-4415-bb98-1878031012e2-kube-api-access-2d8m7\") pod \"cert-manager-cainjector-5c57bbd4fc-4hkl8\" (UID: \"ea879f58-8507-4415-bb98-1878031012e2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4hkl8" Nov 03 19:46:13.527642 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:13.527610 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea879f58-8507-4415-bb98-1878031012e2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4hkl8\" (UID: \"ea879f58-8507-4415-bb98-1878031012e2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4hkl8" Nov 03 19:46:13.527775 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:13.527651 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d8m7\" (UniqueName: \"kubernetes.io/projected/ea879f58-8507-4415-bb98-1878031012e2-kube-api-access-2d8m7\") pod \"cert-manager-cainjector-5c57bbd4fc-4hkl8\" (UID: \"ea879f58-8507-4415-bb98-1878031012e2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4hkl8" Nov 03 19:46:13.536038 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:13.536009 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea879f58-8507-4415-bb98-1878031012e2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4hkl8\" (UID: \"ea879f58-8507-4415-bb98-1878031012e2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4hkl8" Nov 03 19:46:13.536189 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:13.536170 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d8m7\" (UniqueName: \"kubernetes.io/projected/ea879f58-8507-4415-bb98-1878031012e2-kube-api-access-2d8m7\") pod \"cert-manager-cainjector-5c57bbd4fc-4hkl8\" (UID: \"ea879f58-8507-4415-bb98-1878031012e2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4hkl8" Nov 03 19:46:13.654870 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:13.654831 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4hkl8" Nov 03 19:46:13.928006 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:13.927979 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4hkl8"] Nov 03 19:46:13.931910 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:46:13.931882 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea879f58_8507_4415_bb98_1878031012e2.slice/crio-b5f50a0e08fd8e204924200007becb8adb2e9e36122652de3638728a250c1079 WatchSource:0}: Error finding container b5f50a0e08fd8e204924200007becb8adb2e9e36122652de3638728a250c1079: Status 404 returned error can't find the container with id b5f50a0e08fd8e204924200007becb8adb2e9e36122652de3638728a250c1079 Nov 03 19:46:14.061345 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:14.061306 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4hkl8" event={"ID":"ea879f58-8507-4415-bb98-1878031012e2","Type":"ContainerStarted","Data":"b5f50a0e08fd8e204924200007becb8adb2e9e36122652de3638728a250c1079"} Nov 03 19:46:14.578814 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:14.578779 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-8hw7w"] Nov 03 19:46:14.581984 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:14.581967 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8hw7w" Nov 03 19:46:14.586490 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:14.586470 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-vhprp\"" Nov 03 19:46:14.588564 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:14.588542 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 19:46:14.591249 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:14.591228 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:46:14.600853 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:14.600824 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-8hw7w"] Nov 03 19:46:14.738950 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:14.738900 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5k98\" (UniqueName: \"kubernetes.io/projected/0be8edce-f63a-4740-a6d7-dd20f2289dda-kube-api-access-g5k98\") pod \"jobset-operator-674cd59dc5-8hw7w\" (UID: \"0be8edce-f63a-4740-a6d7-dd20f2289dda\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8hw7w" Nov 03 19:46:14.739130 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:14.738974 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0be8edce-f63a-4740-a6d7-dd20f2289dda-tmp\") pod \"jobset-operator-674cd59dc5-8hw7w\" (UID: \"0be8edce-f63a-4740-a6d7-dd20f2289dda\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8hw7w" Nov 03 19:46:14.839708 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:14.839608 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5k98\" (UniqueName: \"kubernetes.io/projected/0be8edce-f63a-4740-a6d7-dd20f2289dda-kube-api-access-g5k98\") pod \"jobset-operator-674cd59dc5-8hw7w\" (UID: \"0be8edce-f63a-4740-a6d7-dd20f2289dda\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8hw7w" Nov 03 19:46:14.839873 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:14.839762 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0be8edce-f63a-4740-a6d7-dd20f2289dda-tmp\") pod \"jobset-operator-674cd59dc5-8hw7w\" (UID: \"0be8edce-f63a-4740-a6d7-dd20f2289dda\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8hw7w" Nov 03 19:46:14.840107 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:14.840087 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0be8edce-f63a-4740-a6d7-dd20f2289dda-tmp\") pod \"jobset-operator-674cd59dc5-8hw7w\" (UID: \"0be8edce-f63a-4740-a6d7-dd20f2289dda\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8hw7w" Nov 03 19:46:14.849148 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:14.849120 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5k98\" (UniqueName: \"kubernetes.io/projected/0be8edce-f63a-4740-a6d7-dd20f2289dda-kube-api-access-g5k98\") pod \"jobset-operator-674cd59dc5-8hw7w\" (UID: \"0be8edce-f63a-4740-a6d7-dd20f2289dda\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8hw7w" Nov 03 19:46:14.898951 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:14.898908 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8hw7w" Nov 03 19:46:15.183031 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:15.182949 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-8hw7w"] Nov 03 19:46:15.188169 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:46:15.188125 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0be8edce_f63a_4740_a6d7_dd20f2289dda.slice/crio-0c9e49fcde5d11a750832bc2d85ad76c4ba64acc5df87ff61e5f8b63e0a8d245 WatchSource:0}: Error finding container 0c9e49fcde5d11a750832bc2d85ad76c4ba64acc5df87ff61e5f8b63e0a8d245: Status 404 returned error can't find the container with id 0c9e49fcde5d11a750832bc2d85ad76c4ba64acc5df87ff61e5f8b63e0a8d245 Nov 03 19:46:15.974098 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:15.974062 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-5rjzk"] Nov 03 19:46:15.983763 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:15.983176 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5rjzk" Nov 03 19:46:15.987801 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:15.987724 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-5rjzk"] Nov 03 19:46:15.991826 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:15.991801 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 19:46:15.992361 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:15.992323 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 19:46:15.993141 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:15.993120 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vw4l4\"" Nov 03 19:46:16.053069 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:16.053025 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfk2h\" (UniqueName: \"kubernetes.io/projected/2e8c119f-1121-4c27-ac2e-f625ff474965-kube-api-access-rfk2h\") pod \"limitador-operator-controller-manager-5778bb4449-5rjzk\" (UID: \"2e8c119f-1121-4c27-ac2e-f625ff474965\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5rjzk" Nov 03 19:46:16.073980 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:16.073943 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8hw7w" event={"ID":"0be8edce-f63a-4740-a6d7-dd20f2289dda","Type":"ContainerStarted","Data":"0c9e49fcde5d11a750832bc2d85ad76c4ba64acc5df87ff61e5f8b63e0a8d245"} Nov 03 19:46:16.154012 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:16.153971 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfk2h\" (UniqueName: \"kubernetes.io/projected/2e8c119f-1121-4c27-ac2e-f625ff474965-kube-api-access-rfk2h\") pod \"limitador-operator-controller-manager-5778bb4449-5rjzk\" (UID: \"2e8c119f-1121-4c27-ac2e-f625ff474965\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5rjzk" Nov 03 19:46:16.164195 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:16.164163 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfk2h\" (UniqueName: \"kubernetes.io/projected/2e8c119f-1121-4c27-ac2e-f625ff474965-kube-api-access-rfk2h\") pod \"limitador-operator-controller-manager-5778bb4449-5rjzk\" (UID: \"2e8c119f-1121-4c27-ac2e-f625ff474965\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5rjzk" Nov 03 19:46:16.302746 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:16.302659 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5rjzk" Nov 03 19:46:16.595491 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:16.595465 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-5rjzk"] Nov 03 19:46:16.597684 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:46:16.597651 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8c119f_1121_4c27_ac2e_f625ff474965.slice/crio-00f5da08383240ee3817c9c4d15dd8bb07ff7e427880de372491100d4145b089 WatchSource:0}: Error finding container 00f5da08383240ee3817c9c4d15dd8bb07ff7e427880de372491100d4145b089: Status 404 returned error can't find the container with id 00f5da08383240ee3817c9c4d15dd8bb07ff7e427880de372491100d4145b089 Nov 03 19:46:17.090227 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:17.090189 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5rjzk" event={"ID":"2e8c119f-1121-4c27-ac2e-f625ff474965","Type":"ContainerStarted","Data":"00f5da08383240ee3817c9c4d15dd8bb07ff7e427880de372491100d4145b089"} Nov 03 19:46:18.099778 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:18.099688 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4hkl8" event={"ID":"ea879f58-8507-4415-bb98-1878031012e2","Type":"ContainerStarted","Data":"a8fa31493192b473202ba2396d98ac6a06431ea228bbad33ff27c7393899333f"} Nov 03 19:46:19.546327 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.545149 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4hkl8" podStartSLOduration=2.770033521 podStartE2EDuration="6.545133261s" totalImagesPullingTime="3.77509974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:18.11987786 +0000 UTC m=+408.082256301" watchObservedRunningTime="2026-11-03 19:46:19.545133261 +0000 UTC m=+409.507511699" Nov 03 19:46:19.547506 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.547469 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h"] Nov 03 19:46:19.553807 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.553708 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.570016 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.569985 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-tqvwk\"" Nov 03 19:46:19.576303 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.575060 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h"] Nov 03 19:46:19.586327 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.586294 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/002888c8-c3b7-4ae8-942a-e16e9263222b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.586586 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.586351 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/002888c8-c3b7-4ae8-942a-e16e9263222b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.586586 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.586387 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/002888c8-c3b7-4ae8-942a-e16e9263222b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.586586 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.586454 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/002888c8-c3b7-4ae8-942a-e16e9263222b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.586586 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.586525 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/002888c8-c3b7-4ae8-942a-e16e9263222b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.586586 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.586566 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn7cm\" (UniqueName: \"kubernetes.io/projected/002888c8-c3b7-4ae8-942a-e16e9263222b-kube-api-access-wn7cm\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.586836 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.586648 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/002888c8-c3b7-4ae8-942a-e16e9263222b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.586836 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.586701 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/002888c8-c3b7-4ae8-942a-e16e9263222b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.586836 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.586740 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/002888c8-c3b7-4ae8-942a-e16e9263222b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.687395 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.687355 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/002888c8-c3b7-4ae8-942a-e16e9263222b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.687596 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.687404 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/002888c8-c3b7-4ae8-942a-e16e9263222b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.687596 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.687431 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/002888c8-c3b7-4ae8-942a-e16e9263222b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.687596 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.687464 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/002888c8-c3b7-4ae8-942a-e16e9263222b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.687596 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.687509 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/002888c8-c3b7-4ae8-942a-e16e9263222b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.687596 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.687558 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7cm\" (UniqueName: \"kubernetes.io/projected/002888c8-c3b7-4ae8-942a-e16e9263222b-kube-api-access-wn7cm\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.687891 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.687615 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/002888c8-c3b7-4ae8-942a-e16e9263222b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.687891 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.687657 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/002888c8-c3b7-4ae8-942a-e16e9263222b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.687891 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.687680 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/002888c8-c3b7-4ae8-942a-e16e9263222b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.687891 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.687819 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/002888c8-c3b7-4ae8-942a-e16e9263222b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.688096 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.687886 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/002888c8-c3b7-4ae8-942a-e16e9263222b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.688514 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.688314 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/002888c8-c3b7-4ae8-942a-e16e9263222b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.688514 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.688460 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/002888c8-c3b7-4ae8-942a-e16e9263222b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.688691 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.688527 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/002888c8-c3b7-4ae8-942a-e16e9263222b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.690466 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.690446 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/002888c8-c3b7-4ae8-942a-e16e9263222b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.693033 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.693011 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/002888c8-c3b7-4ae8-942a-e16e9263222b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.695797 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.695770 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/002888c8-c3b7-4ae8-942a-e16e9263222b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.696198 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.696176 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7cm\" (UniqueName: \"kubernetes.io/projected/002888c8-c3b7-4ae8-942a-e16e9263222b-kube-api-access-wn7cm\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-pv54h\" (UID: \"002888c8-c3b7-4ae8-942a-e16e9263222b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:19.874334 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:19.874304 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:20.114633 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:20.114505 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5rjzk" event={"ID":"2e8c119f-1121-4c27-ac2e-f625ff474965","Type":"ContainerStarted","Data":"b603b66416347715be324d7dc646f7064c3ffde1db12aea4562bb37bade3c891"} Nov 03 19:46:20.114789 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:20.114708 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5rjzk" Nov 03 19:46:20.116399 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:20.116370 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8hw7w" event={"ID":"0be8edce-f63a-4740-a6d7-dd20f2289dda","Type":"ContainerStarted","Data":"625cd5f8f0dd40b517aaf93ba54db847fa4239f5906d441e6c5e4648f08eb6d1"} Nov 03 19:46:20.132495 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:20.132441 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5rjzk" podStartSLOduration=1.8671097840000002 podStartE2EDuration="5.132426028s" totalImagesPullingTime="3.265316244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:20.131139364 +0000 UTC m=+410.093517804" watchObservedRunningTime="2026-11-03 19:46:20.132426028 +0000 UTC m=+410.094804467" Nov 03 19:46:20.150312 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:20.150248 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8hw7w" podStartSLOduration=1.791503332 podStartE2EDuration="6.150228577s" totalImagesPullingTime="4.358725245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:20.147872345 +0000 UTC m=+410.110250808" watchObservedRunningTime="2026-11-03 19:46:20.150228577 +0000 UTC m=+410.112607015" Nov 03 19:46:20.171034 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:20.171002 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h"] Nov 03 19:46:20.173887 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:46:20.173845 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002888c8_c3b7_4ae8_942a_e16e9263222b.slice/crio-ff9da7ed45aeef6c7f1792ebdc0a1aef1cb912075846e019e578c32aa588d3b5 WatchSource:0}: Error finding container ff9da7ed45aeef6c7f1792ebdc0a1aef1cb912075846e019e578c32aa588d3b5: Status 404 returned error can't find the container with id ff9da7ed45aeef6c7f1792ebdc0a1aef1cb912075846e019e578c32aa588d3b5 Nov 03 19:46:20.178319 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:20.178277 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:20.178442 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:20.178361 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:20.178442 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:20.178401 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:46:21.122493 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:21.122457 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" event={"ID":"002888c8-c3b7-4ae8-942a-e16e9263222b","Type":"ContainerStarted","Data":"4dd426be4d7c02a06cf6b0552052a97ed800d0e9dca4b539a10317d5f3bbb1e2"} Nov 03 19:46:21.123375 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:21.122523 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" event={"ID":"002888c8-c3b7-4ae8-942a-e16e9263222b","Type":"ContainerStarted","Data":"ff9da7ed45aeef6c7f1792ebdc0a1aef1cb912075846e019e578c32aa588d3b5"} Nov 03 19:46:21.144012 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:21.143962 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" podStartSLOduration=2.143949294 podStartE2EDuration="2.143949294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:21.142471979 +0000 UTC m=+411.104850417" watchObservedRunningTime="2026-11-03 19:46:21.143949294 +0000 UTC m=+411.106327787" Nov 03 19:46:21.874789 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:21.874750 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:21.879639 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:21.879611 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:22.127330 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:22.127249 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:22.128230 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:22.128212 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-pv54h" Nov 03 19:46:31.125432 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:31.125395 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5rjzk" Nov 03 19:46:35.053953 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:46:35.053910 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:46:57.018259 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.018215 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c"] Nov 03 19:46:57.025760 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.025732 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" Nov 03 19:46:57.029228 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.029198 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c"] Nov 03 19:46:57.030564 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.030521 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 19:46:57.030564 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.030555 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 19:46:57.030768 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.030563 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 19:46:57.030768 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.030707 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-6sb9b\"" Nov 03 19:46:57.100449 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.100408 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/225423cb-b6d0-4ea9-a618-1815a9216d3b-metrics-certs\") pod \"jobset-controller-manager-5db7bf684b-r226c\" (UID: \"225423cb-b6d0-4ea9-a618-1815a9216d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" Nov 03 19:46:57.100675 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.100466 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/225423cb-b6d0-4ea9-a618-1815a9216d3b-manager-config\") pod \"jobset-controller-manager-5db7bf684b-r226c\" (UID: \"225423cb-b6d0-4ea9-a618-1815a9216d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" Nov 03 19:46:57.100675 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.100610 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9sj2\" (UniqueName: \"kubernetes.io/projected/225423cb-b6d0-4ea9-a618-1815a9216d3b-kube-api-access-j9sj2\") pod \"jobset-controller-manager-5db7bf684b-r226c\" (UID: \"225423cb-b6d0-4ea9-a618-1815a9216d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" Nov 03 19:46:57.100675 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.100648 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/225423cb-b6d0-4ea9-a618-1815a9216d3b-cert\") pod \"jobset-controller-manager-5db7bf684b-r226c\" (UID: \"225423cb-b6d0-4ea9-a618-1815a9216d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" Nov 03 19:46:57.201580 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.201512 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/225423cb-b6d0-4ea9-a618-1815a9216d3b-manager-config\") pod \"jobset-controller-manager-5db7bf684b-r226c\" (UID: \"225423cb-b6d0-4ea9-a618-1815a9216d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" Nov 03 19:46:57.201777 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.201618 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9sj2\" (UniqueName: \"kubernetes.io/projected/225423cb-b6d0-4ea9-a618-1815a9216d3b-kube-api-access-j9sj2\") pod \"jobset-controller-manager-5db7bf684b-r226c\" (UID: \"225423cb-b6d0-4ea9-a618-1815a9216d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" Nov 03 19:46:57.201777 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.201645 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/225423cb-b6d0-4ea9-a618-1815a9216d3b-cert\") pod \"jobset-controller-manager-5db7bf684b-r226c\" (UID: \"225423cb-b6d0-4ea9-a618-1815a9216d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" Nov 03 19:46:57.201777 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.201705 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/225423cb-b6d0-4ea9-a618-1815a9216d3b-metrics-certs\") pod \"jobset-controller-manager-5db7bf684b-r226c\" (UID: \"225423cb-b6d0-4ea9-a618-1815a9216d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" Nov 03 19:46:57.202172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.202147 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/225423cb-b6d0-4ea9-a618-1815a9216d3b-manager-config\") pod \"jobset-controller-manager-5db7bf684b-r226c\" (UID: \"225423cb-b6d0-4ea9-a618-1815a9216d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" Nov 03 19:46:57.204213 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.204193 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/225423cb-b6d0-4ea9-a618-1815a9216d3b-metrics-certs\") pod \"jobset-controller-manager-5db7bf684b-r226c\" (UID: \"225423cb-b6d0-4ea9-a618-1815a9216d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" Nov 03 19:46:57.204294 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.204231 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/225423cb-b6d0-4ea9-a618-1815a9216d3b-cert\") pod \"jobset-controller-manager-5db7bf684b-r226c\" (UID: \"225423cb-b6d0-4ea9-a618-1815a9216d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" Nov 03 19:46:57.209599 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.209582 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9sj2\" (UniqueName: \"kubernetes.io/projected/225423cb-b6d0-4ea9-a618-1815a9216d3b-kube-api-access-j9sj2\") pod \"jobset-controller-manager-5db7bf684b-r226c\" (UID: \"225423cb-b6d0-4ea9-a618-1815a9216d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" Nov 03 19:46:57.339731 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.339650 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" Nov 03 19:46:57.608829 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:57.608806 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c"] Nov 03 19:46:57.611099 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:46:57.611066 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225423cb_b6d0_4ea9_a618_1815a9216d3b.slice/crio-67072a2785fde62a1ceb2382a601f40dd6f7ab40aa0eab03ffe0e29be6cd429f WatchSource:0}: Error finding container 67072a2785fde62a1ceb2382a601f40dd6f7ab40aa0eab03ffe0e29be6cd429f: Status 404 returned error can't find the container with id 67072a2785fde62a1ceb2382a601f40dd6f7ab40aa0eab03ffe0e29be6cd429f Nov 03 19:46:58.287215 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:46:58.287180 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" event={"ID":"225423cb-b6d0-4ea9-a618-1815a9216d3b","Type":"ContainerStarted","Data":"67072a2785fde62a1ceb2382a601f40dd6f7ab40aa0eab03ffe0e29be6cd429f"} Nov 03 19:47:02.307449 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:47:02.307408 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" event={"ID":"225423cb-b6d0-4ea9-a618-1815a9216d3b","Type":"ContainerStarted","Data":"6f0dc5c4e2d64e92530c8bac6c5c696d5e914fbabdb5a1c1cb1c3bd66d1c8817"} Nov 03 19:47:02.307981 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:47:02.307523 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" Nov 03 19:47:02.324611 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:47:02.324560 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" podStartSLOduration=1.376661654 podStartE2EDuration="5.324546938s" totalImagesPullingTime="3.947885284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:02.323942029 +0000 UTC m=+452.286320468" watchObservedRunningTime="2026-11-03 19:47:02.324546938 +0000 UTC m=+452.286925368" Nov 03 19:47:05.054377 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:47:05.054340 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:47:13.317960 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:47:13.317929 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5db7bf684b-r226c" Nov 03 19:47:35.054868 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:47:35.054831 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:48:05.054083 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:48:05.054043 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:48:35.054283 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:48:35.054244 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:49:05.054985 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:49:05.054938 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:49:30.225917 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:49:30.225830 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 19:49:30.228427 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:49:30.226700 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 19:49:30.232096 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:49:30.232077 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 19:49:30.233223 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:49:30.233201 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 19:49:35.055993 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:49:35.055952 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:50:05.055030 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:50:05.054985 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:50:35.055572 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:50:35.055508 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:51:05.055601 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:51:05.055555 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:51:35.055528 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:51:35.055490 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:52:05.055926 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:52:05.055889 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:52:35.055198 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:52:35.055106 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:53:05.055081 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:53:05.055038 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:53:35.055465 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:53:35.055399 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:54:05.056149 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:54:05.056106 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:54:30.260837 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:54:30.260725 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 19:54:30.264892 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:54:30.262377 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 19:54:30.266396 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:54:30.266377 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 19:54:30.267480 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:54:30.267463 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 19:54:35.055621 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:54:35.055582 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:55:05.055730 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:55:05.055684 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:55:35.054825 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:55:35.054743 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:56:05.055069 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:56:05.055031 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:56:35.055665 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:56:35.055624 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:56:57.756608 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:56:57.756566 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-grrsr"] Nov 03 19:56:57.760081 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:56:57.760057 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-grrsr" Nov 03 19:56:57.767198 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:56:57.767157 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-grrsr"] Nov 03 19:56:57.768335 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:56:57.768302 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-n72st\"" Nov 03 19:56:57.853327 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:56:57.853278 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bjh5\" (UniqueName: \"kubernetes.io/projected/f342b227-efec-4e32-b7aa-9e2da2f17d57-kube-api-access-5bjh5\") pod \"authorino-57f7f8659f-grrsr\" (UID: \"f342b227-efec-4e32-b7aa-9e2da2f17d57\") " pod="kuadrant-system/authorino-57f7f8659f-grrsr" Nov 03 19:56:57.954593 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:56:57.954514 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjh5\" (UniqueName: \"kubernetes.io/projected/f342b227-efec-4e32-b7aa-9e2da2f17d57-kube-api-access-5bjh5\") pod \"authorino-57f7f8659f-grrsr\" (UID: \"f342b227-efec-4e32-b7aa-9e2da2f17d57\") " pod="kuadrant-system/authorino-57f7f8659f-grrsr" Nov 03 19:56:57.962900 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:56:57.962867 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bjh5\" (UniqueName: \"kubernetes.io/projected/f342b227-efec-4e32-b7aa-9e2da2f17d57-kube-api-access-5bjh5\") pod \"authorino-57f7f8659f-grrsr\" (UID: \"f342b227-efec-4e32-b7aa-9e2da2f17d57\") " pod="kuadrant-system/authorino-57f7f8659f-grrsr" Nov 03 19:56:58.073657 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:56:58.073569 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-grrsr" Nov 03 19:56:58.351464 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:56:58.351436 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-grrsr"] Nov 03 19:56:58.354217 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:56:58.354189 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf342b227_efec_4e32_b7aa_9e2da2f17d57.slice/crio-52bfd58d9b30ae49e32617aeac1a2dea50a8ec8231ff52148db153ae375de18b WatchSource:0}: Error finding container 52bfd58d9b30ae49e32617aeac1a2dea50a8ec8231ff52148db153ae375de18b: Status 404 returned error can't find the container with id 52bfd58d9b30ae49e32617aeac1a2dea50a8ec8231ff52148db153ae375de18b Nov 03 19:56:58.355556 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:56:58.355518 5111 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:56:58.966056 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:56:58.966021 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-grrsr" event={"ID":"f342b227-efec-4e32-b7aa-9e2da2f17d57","Type":"ContainerStarted","Data":"52bfd58d9b30ae49e32617aeac1a2dea50a8ec8231ff52148db153ae375de18b"} Nov 03 19:57:01.289479 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:01.289442 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-grrsr"] Nov 03 19:57:03.989750 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:03.989713 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-grrsr" event={"ID":"f342b227-efec-4e32-b7aa-9e2da2f17d57","Type":"ContainerStarted","Data":"ece21bba6439fc29578010b058b99f0f9977b9744b0652e05c4882f5e65b3b85"} Nov 03 19:57:03.990184 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:03.989796 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-grrsr" podUID="f342b227-efec-4e32-b7aa-9e2da2f17d57" containerName="authorino" containerID="cri-o://ece21bba6439fc29578010b058b99f0f9977b9744b0652e05c4882f5e65b3b85" gracePeriod=30 Nov 03 19:57:04.004276 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:04.004233 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-grrsr" podStartSLOduration=2.129617291 podStartE2EDuration="7.004220084s" totalImagesPullingTime="4.874602793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:04.002720159 +0000 UTC m=+1053.965098599" watchObservedRunningTime="2026-11-03 19:57:04.004220084 +0000 UTC m=+1053.966598523" Nov 03 19:57:04.229228 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:04.229202 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-grrsr" Nov 03 19:57:04.307303 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:04.307228 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f342b227-efec-4e32-b7aa-9e2da2f17d57-kube-api-access-5bjh5\" (UniqueName: \"kubernetes.io/projected/f342b227-efec-4e32-b7aa-9e2da2f17d57-kube-api-access-5bjh5\") pod \"f342b227-efec-4e32-b7aa-9e2da2f17d57\" (UID: \"f342b227-efec-4e32-b7aa-9e2da2f17d57\") " Nov 03 19:57:04.309469 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:04.309441 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f342b227-efec-4e32-b7aa-9e2da2f17d57-kube-api-access-5bjh5" pod "f342b227-efec-4e32-b7aa-9e2da2f17d57" (UID: "f342b227-efec-4e32-b7aa-9e2da2f17d57"). InnerVolumeSpecName "kube-api-access-5bjh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:04.408414 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:04.408376 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bjh5\" (UniqueName: \"kubernetes.io/projected/f342b227-efec-4e32-b7aa-9e2da2f17d57-kube-api-access-5bjh5\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:04.995583 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:04.995549 5111 generic.go:356] "Generic (PLEG): container finished" podID="f342b227-efec-4e32-b7aa-9e2da2f17d57" containerID="ece21bba6439fc29578010b058b99f0f9977b9744b0652e05c4882f5e65b3b85" exitCode=0 Nov 03 19:57:04.996006 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:04.995590 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-grrsr" Nov 03 19:57:04.996006 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:04.995624 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-grrsr" event={"ID":"f342b227-efec-4e32-b7aa-9e2da2f17d57","Type":"ContainerDied","Data":"ece21bba6439fc29578010b058b99f0f9977b9744b0652e05c4882f5e65b3b85"} Nov 03 19:57:04.996006 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:04.995666 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-grrsr" event={"ID":"f342b227-efec-4e32-b7aa-9e2da2f17d57","Type":"ContainerDied","Data":"52bfd58d9b30ae49e32617aeac1a2dea50a8ec8231ff52148db153ae375de18b"} Nov 03 19:57:04.996006 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:04.995683 5111 scope.go:122] "RemoveContainer" containerID="ece21bba6439fc29578010b058b99f0f9977b9744b0652e05c4882f5e65b3b85" Nov 03 19:57:05.004432 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:05.004415 5111 scope.go:122] "RemoveContainer" containerID="ece21bba6439fc29578010b058b99f0f9977b9744b0652e05c4882f5e65b3b85" Nov 03 19:57:05.004725 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:57:05.004704 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece21bba6439fc29578010b058b99f0f9977b9744b0652e05c4882f5e65b3b85\": container with ID starting with ece21bba6439fc29578010b058b99f0f9977b9744b0652e05c4882f5e65b3b85 not found: ID does not exist" containerID="ece21bba6439fc29578010b058b99f0f9977b9744b0652e05c4882f5e65b3b85" Nov 03 19:57:05.004808 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:05.004737 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece21bba6439fc29578010b058b99f0f9977b9744b0652e05c4882f5e65b3b85"} err="failed to get container status \"ece21bba6439fc29578010b058b99f0f9977b9744b0652e05c4882f5e65b3b85\": rpc error: code = NotFound desc = could not find container \"ece21bba6439fc29578010b058b99f0f9977b9744b0652e05c4882f5e65b3b85\": container with ID starting with ece21bba6439fc29578010b058b99f0f9977b9744b0652e05c4882f5e65b3b85 not found: ID does not exist" Nov 03 19:57:05.016090 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:05.016067 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-grrsr"] Nov 03 19:57:05.017962 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:05.017941 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-grrsr"] Nov 03 19:57:05.054708 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:57:05.054677 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:57:06.180751 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:06.180722 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f342b227-efec-4e32-b7aa-9e2da2f17d57" path="/var/lib/kubelet/pods/f342b227-efec-4e32-b7aa-9e2da2f17d57/volumes" Nov 03 19:57:35.055951 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:57:35.055913 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:57:38.304591 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.304556 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-5v9gz"] Nov 03 19:57:38.305072 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.304980 5111 cpu_manager.go:414] "containerMap: removing container" podUID="f342b227-efec-4e32-b7aa-9e2da2f17d57" containerName="authorino" Nov 03 19:57:38.305072 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.304993 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f342b227-efec-4e32-b7aa-9e2da2f17d57" containerName="authorino" Nov 03 19:57:38.305072 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.305054 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="f342b227-efec-4e32-b7aa-9e2da2f17d57" containerName="authorino" Nov 03 19:57:38.309257 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.309235 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-5v9gz" Nov 03 19:57:38.317089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.316805 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 19:57:38.317089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.316900 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 19:57:38.318000 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.317976 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 19:57:38.318116 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.318026 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 19:57:38.318116 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.317977 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 19:57:38.318319 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.318301 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6dlrg\"" Nov 03 19:57:38.320278 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.320257 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-5v9gz"] Nov 03 19:57:38.391522 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.391486 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xr4z\" (UniqueName: \"kubernetes.io/projected/6800e09e-2aee-4032-b28d-c4397c107f7d-kube-api-access-7xr4z\") pod \"maas-controller-579cf7c6d-5v9gz\" (UID: \"6800e09e-2aee-4032-b28d-c4397c107f7d\") " pod="opendatahub/maas-controller-579cf7c6d-5v9gz" Nov 03 19:57:38.391714 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.391559 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-webhook-cert\") pod \"maas-controller-579cf7c6d-5v9gz\" (UID: \"6800e09e-2aee-4032-b28d-c4397c107f7d\") " pod="opendatahub/maas-controller-579cf7c6d-5v9gz" Nov 03 19:57:38.391714 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.391666 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-metrics-certs\") pod \"maas-controller-579cf7c6d-5v9gz\" (UID: \"6800e09e-2aee-4032-b28d-c4397c107f7d\") " pod="opendatahub/maas-controller-579cf7c6d-5v9gz" Nov 03 19:57:38.492762 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.492721 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-metrics-certs\") pod \"maas-controller-579cf7c6d-5v9gz\" (UID: \"6800e09e-2aee-4032-b28d-c4397c107f7d\") " pod="opendatahub/maas-controller-579cf7c6d-5v9gz" Nov 03 19:57:38.492937 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.492809 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xr4z\" (UniqueName: \"kubernetes.io/projected/6800e09e-2aee-4032-b28d-c4397c107f7d-kube-api-access-7xr4z\") pod \"maas-controller-579cf7c6d-5v9gz\" (UID: \"6800e09e-2aee-4032-b28d-c4397c107f7d\") " pod="opendatahub/maas-controller-579cf7c6d-5v9gz" Nov 03 19:57:38.492937 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.492831 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-webhook-cert\") pod \"maas-controller-579cf7c6d-5v9gz\" (UID: \"6800e09e-2aee-4032-b28d-c4397c107f7d\") " pod="opendatahub/maas-controller-579cf7c6d-5v9gz" Nov 03 19:57:38.492937 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:57:38.492919 5111 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Nov 03 19:57:38.493041 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:57:38.492980 5111 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-webhook-cert podName:6800e09e-2aee-4032-b28d-c4397c107f7d nodeName:}" failed. No retries permitted until 2026-11-03 19:57:38.992964805 +0000 UTC m=+1088.955343221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-webhook-cert") pod "maas-controller-579cf7c6d-5v9gz" (UID: "6800e09e-2aee-4032-b28d-c4397c107f7d") : secret "maas-controller-webhook-cert" not found Nov 03 19:57:38.495192 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.495171 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-metrics-certs\") pod \"maas-controller-579cf7c6d-5v9gz\" (UID: \"6800e09e-2aee-4032-b28d-c4397c107f7d\") " pod="opendatahub/maas-controller-579cf7c6d-5v9gz" Nov 03 19:57:38.501908 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.501883 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xr4z\" (UniqueName: \"kubernetes.io/projected/6800e09e-2aee-4032-b28d-c4397c107f7d-kube-api-access-7xr4z\") pod \"maas-controller-579cf7c6d-5v9gz\" (UID: \"6800e09e-2aee-4032-b28d-c4397c107f7d\") " pod="opendatahub/maas-controller-579cf7c6d-5v9gz" Nov 03 19:57:38.615103 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.615025 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65f5dc4b79-ltr7j"] Nov 03 19:57:38.618657 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.618621 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" Nov 03 19:57:38.629007 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.628984 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f5dc4b79-ltr7j"] Nov 03 19:57:38.694917 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.694883 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02f9d4b8-695a-4049-8bdb-1d62792ed657-webhook-cert\") pod \"maas-controller-65f5dc4b79-ltr7j\" (UID: \"02f9d4b8-695a-4049-8bdb-1d62792ed657\") " pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" Nov 03 19:57:38.695092 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.694930 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f9d4b8-695a-4049-8bdb-1d62792ed657-metrics-certs\") pod \"maas-controller-65f5dc4b79-ltr7j\" (UID: \"02f9d4b8-695a-4049-8bdb-1d62792ed657\") " pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" Nov 03 19:57:38.695092 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.695053 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfczw\" (UniqueName: \"kubernetes.io/projected/02f9d4b8-695a-4049-8bdb-1d62792ed657-kube-api-access-bfczw\") pod \"maas-controller-65f5dc4b79-ltr7j\" (UID: \"02f9d4b8-695a-4049-8bdb-1d62792ed657\") " pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" Nov 03 19:57:38.796331 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.796294 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfczw\" (UniqueName: \"kubernetes.io/projected/02f9d4b8-695a-4049-8bdb-1d62792ed657-kube-api-access-bfczw\") pod \"maas-controller-65f5dc4b79-ltr7j\" (UID: \"02f9d4b8-695a-4049-8bdb-1d62792ed657\") " pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" Nov 03 19:57:38.796563 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.796356 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02f9d4b8-695a-4049-8bdb-1d62792ed657-webhook-cert\") pod \"maas-controller-65f5dc4b79-ltr7j\" (UID: \"02f9d4b8-695a-4049-8bdb-1d62792ed657\") " pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" Nov 03 19:57:38.796563 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.796401 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f9d4b8-695a-4049-8bdb-1d62792ed657-metrics-certs\") pod \"maas-controller-65f5dc4b79-ltr7j\" (UID: \"02f9d4b8-695a-4049-8bdb-1d62792ed657\") " pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" Nov 03 19:57:38.799014 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.798988 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f9d4b8-695a-4049-8bdb-1d62792ed657-metrics-certs\") pod \"maas-controller-65f5dc4b79-ltr7j\" (UID: \"02f9d4b8-695a-4049-8bdb-1d62792ed657\") " pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" Nov 03 19:57:38.799088 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.799036 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02f9d4b8-695a-4049-8bdb-1d62792ed657-webhook-cert\") pod \"maas-controller-65f5dc4b79-ltr7j\" (UID: \"02f9d4b8-695a-4049-8bdb-1d62792ed657\") " pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" Nov 03 19:57:38.803872 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.803854 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfczw\" (UniqueName: \"kubernetes.io/projected/02f9d4b8-695a-4049-8bdb-1d62792ed657-kube-api-access-bfczw\") pod \"maas-controller-65f5dc4b79-ltr7j\" (UID: \"02f9d4b8-695a-4049-8bdb-1d62792ed657\") " pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" Nov 03 19:57:38.932345 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.932306 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" Nov 03 19:57:38.998413 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:38.998372 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-webhook-cert\") pod \"maas-controller-579cf7c6d-5v9gz\" (UID: \"6800e09e-2aee-4032-b28d-c4397c107f7d\") " pod="opendatahub/maas-controller-579cf7c6d-5v9gz" Nov 03 19:57:39.001444 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:39.001419 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-webhook-cert\") pod \"maas-controller-579cf7c6d-5v9gz\" (UID: \"6800e09e-2aee-4032-b28d-c4397c107f7d\") " pod="opendatahub/maas-controller-579cf7c6d-5v9gz" Nov 03 19:57:39.205903 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:39.205815 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f5dc4b79-ltr7j"] Nov 03 19:57:39.208917 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:57:39.208886 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f9d4b8_695a_4049_8bdb_1d62792ed657.slice/crio-e62feb62c1c93b2fc88ba6fdbcd97f1cfc90f8c25a9b7b56bba77f848c545b49 WatchSource:0}: Error finding container e62feb62c1c93b2fc88ba6fdbcd97f1cfc90f8c25a9b7b56bba77f848c545b49: Status 404 returned error can't find the container with id e62feb62c1c93b2fc88ba6fdbcd97f1cfc90f8c25a9b7b56bba77f848c545b49 Nov 03 19:57:39.235483 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:39.235451 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-5v9gz" Nov 03 19:57:39.510573 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:39.510547 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-5v9gz"] Nov 03 19:57:39.512881 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:57:39.512855 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6800e09e_2aee_4032_b28d_c4397c107f7d.slice/crio-ede312f357edab72381d2c02b152289adc14d02c088d9c069e0d8820816d79bd WatchSource:0}: Error finding container ede312f357edab72381d2c02b152289adc14d02c088d9c069e0d8820816d79bd: Status 404 returned error can't find the container with id ede312f357edab72381d2c02b152289adc14d02c088d9c069e0d8820816d79bd Nov 03 19:57:40.157447 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:40.157401 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" event={"ID":"02f9d4b8-695a-4049-8bdb-1d62792ed657","Type":"ContainerStarted","Data":"e62feb62c1c93b2fc88ba6fdbcd97f1cfc90f8c25a9b7b56bba77f848c545b49"} Nov 03 19:57:40.159207 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:40.159174 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-5v9gz" event={"ID":"6800e09e-2aee-4032-b28d-c4397c107f7d","Type":"ContainerStarted","Data":"ede312f357edab72381d2c02b152289adc14d02c088d9c069e0d8820816d79bd"} Nov 03 19:57:43.179512 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:43.179473 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" event={"ID":"02f9d4b8-695a-4049-8bdb-1d62792ed657","Type":"ContainerStarted","Data":"6ac57524f9fcf7cfc4531cb12ef1045cb6cd7cb3d6cb6ef2ef5e15f297436d28"} Nov 03 19:57:43.180027 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:43.179558 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" Nov 03 19:57:43.181203 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:43.181178 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-5v9gz" event={"ID":"6800e09e-2aee-4032-b28d-c4397c107f7d","Type":"ContainerStarted","Data":"5e6645bc8d638018ba8e1f5e7328401330dc1c788f8b3bfc74c6edff9377aa5f"} Nov 03 19:57:43.181305 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:43.181243 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-5v9gz" Nov 03 19:57:43.200414 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:43.200367 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" podStartSLOduration=2.248522559 podStartE2EDuration="5.200353971s" totalImagesPullingTime="2.951831412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:43.199412735 +0000 UTC m=+1093.161791175" watchObservedRunningTime="2026-11-03 19:57:43.200353971 +0000 UTC m=+1093.162732410" Nov 03 19:57:43.217738 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:43.217670 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-5v9gz" podStartSLOduration=2.406439241 podStartE2EDuration="5.217654514s" totalImagesPullingTime="2.811215273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:43.217128272 +0000 UTC m=+1093.179506733" watchObservedRunningTime="2026-11-03 19:57:43.217654514 +0000 UTC m=+1093.180032953" Nov 03 19:57:54.189197 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:54.189163 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" Nov 03 19:57:54.189726 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:54.189239 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-5v9gz" Nov 03 19:57:54.247019 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:54.246975 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-5v9gz"] Nov 03 19:57:54.247227 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:54.247194 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-5v9gz" podUID="6800e09e-2aee-4032-b28d-c4397c107f7d" containerName="manager" containerID="cri-o://5e6645bc8d638018ba8e1f5e7328401330dc1c788f8b3bfc74c6edff9377aa5f" gracePeriod=30 Nov 03 19:57:54.490904 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:54.490879 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-5v9gz" Nov 03 19:57:54.537349 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:54.537325 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-webhook-cert\") pod \"6800e09e-2aee-4032-b28d-c4397c107f7d\" (UID: \"6800e09e-2aee-4032-b28d-c4397c107f7d\") " Nov 03 19:57:54.537511 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:54.537440 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6800e09e-2aee-4032-b28d-c4397c107f7d-kube-api-access-7xr4z\" (UniqueName: \"kubernetes.io/projected/6800e09e-2aee-4032-b28d-c4397c107f7d-kube-api-access-7xr4z\") pod \"6800e09e-2aee-4032-b28d-c4397c107f7d\" (UID: \"6800e09e-2aee-4032-b28d-c4397c107f7d\") " Nov 03 19:57:54.537511 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:54.537489 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-metrics-certs\") pod \"6800e09e-2aee-4032-b28d-c4397c107f7d\" (UID: \"6800e09e-2aee-4032-b28d-c4397c107f7d\") " Nov 03 19:57:54.539619 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:54.539588 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-webhook-cert" pod "6800e09e-2aee-4032-b28d-c4397c107f7d" (UID: "6800e09e-2aee-4032-b28d-c4397c107f7d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:57:54.539733 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:54.539650 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-metrics-certs" pod "6800e09e-2aee-4032-b28d-c4397c107f7d" (UID: "6800e09e-2aee-4032-b28d-c4397c107f7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:57:54.539733 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:54.539650 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6800e09e-2aee-4032-b28d-c4397c107f7d-kube-api-access-7xr4z" pod "6800e09e-2aee-4032-b28d-c4397c107f7d" (UID: "6800e09e-2aee-4032-b28d-c4397c107f7d"). InnerVolumeSpecName "kube-api-access-7xr4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:54.638515 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:54.638480 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xr4z\" (UniqueName: \"kubernetes.io/projected/6800e09e-2aee-4032-b28d-c4397c107f7d-kube-api-access-7xr4z\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:54.638515 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:54.638514 5111 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-metrics-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:54.638515 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:54.638524 5111 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6800e09e-2aee-4032-b28d-c4397c107f7d-webhook-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.238008 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:55.237971 5111 generic.go:356] "Generic (PLEG): container finished" podID="6800e09e-2aee-4032-b28d-c4397c107f7d" containerID="5e6645bc8d638018ba8e1f5e7328401330dc1c788f8b3bfc74c6edff9377aa5f" exitCode=0 Nov 03 19:57:55.238457 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:55.238028 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-5v9gz" Nov 03 19:57:55.238457 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:55.238059 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-5v9gz" event={"ID":"6800e09e-2aee-4032-b28d-c4397c107f7d","Type":"ContainerDied","Data":"5e6645bc8d638018ba8e1f5e7328401330dc1c788f8b3bfc74c6edff9377aa5f"} Nov 03 19:57:55.238457 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:55.238108 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-5v9gz" event={"ID":"6800e09e-2aee-4032-b28d-c4397c107f7d","Type":"ContainerDied","Data":"ede312f357edab72381d2c02b152289adc14d02c088d9c069e0d8820816d79bd"} Nov 03 19:57:55.238457 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:55.238130 5111 scope.go:122] "RemoveContainer" containerID="5e6645bc8d638018ba8e1f5e7328401330dc1c788f8b3bfc74c6edff9377aa5f" Nov 03 19:57:55.248104 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:55.247950 5111 scope.go:122] "RemoveContainer" containerID="5e6645bc8d638018ba8e1f5e7328401330dc1c788f8b3bfc74c6edff9377aa5f" Nov 03 19:57:55.248196 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:57:55.248178 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6645bc8d638018ba8e1f5e7328401330dc1c788f8b3bfc74c6edff9377aa5f\": container with ID starting with 5e6645bc8d638018ba8e1f5e7328401330dc1c788f8b3bfc74c6edff9377aa5f not found: ID does not exist" containerID="5e6645bc8d638018ba8e1f5e7328401330dc1c788f8b3bfc74c6edff9377aa5f" Nov 03 19:57:55.248246 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:55.248205 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6645bc8d638018ba8e1f5e7328401330dc1c788f8b3bfc74c6edff9377aa5f"} err="failed to get container status \"5e6645bc8d638018ba8e1f5e7328401330dc1c788f8b3bfc74c6edff9377aa5f\": rpc error: code = NotFound desc = could not find container \"5e6645bc8d638018ba8e1f5e7328401330dc1c788f8b3bfc74c6edff9377aa5f\": container with ID starting with 5e6645bc8d638018ba8e1f5e7328401330dc1c788f8b3bfc74c6edff9377aa5f not found: ID does not exist" Nov 03 19:57:55.259285 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:55.259264 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-5v9gz"] Nov 03 19:57:55.263216 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:55.263194 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-5v9gz"] Nov 03 19:57:56.180591 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:57:56.180560 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6800e09e-2aee-4032-b28d-c4397c107f7d" path="/var/lib/kubelet/pods/6800e09e-2aee-4032-b28d-c4397c107f7d/volumes" Nov 03 19:58:05.055068 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:58:05.055027 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:58:26.076922 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.076888 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ms8vz"] Nov 03 19:58:26.077346 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.077328 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6800e09e-2aee-4032-b28d-c4397c107f7d" containerName="manager" Nov 03 19:58:26.077398 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.077349 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6800e09e-2aee-4032-b28d-c4397c107f7d" containerName="manager" Nov 03 19:58:26.077444 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.077435 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6800e09e-2aee-4032-b28d-c4397c107f7d" containerName="manager" Nov 03 19:58:26.080161 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.080143 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ms8vz" Nov 03 19:58:26.086341 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.086319 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-n72st\"" Nov 03 19:58:26.087219 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.087154 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ms8vz"] Nov 03 19:58:26.131865 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.131830 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hglq9\" (UniqueName: \"kubernetes.io/projected/9682e516-2495-4ec2-8aaa-5e0514e0ce65-kube-api-access-hglq9\") pod \"authorino-6f6944bcdd-ms8vz\" (UID: \"9682e516-2495-4ec2-8aaa-5e0514e0ce65\") " pod="kuadrant-system/authorino-6f6944bcdd-ms8vz" Nov 03 19:58:26.162698 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.162613 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ms8vz"] Nov 03 19:58:26.162882 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:58:26.162863 5111 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hglq9], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-ms8vz" podUID="9682e516-2495-4ec2-8aaa-5e0514e0ce65" Nov 03 19:58:26.192419 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.192388 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nhxns"] Nov 03 19:58:26.195927 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.195909 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-nhxns" Nov 03 19:58:26.203793 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.203751 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 19:58:26.205423 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.205400 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nhxns"] Nov 03 19:58:26.232398 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.232363 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e19f3dad-1f20-4cb5-8940-1408e9a22ae3-tls-cert\") pod \"authorino-6889cbd8c4-nhxns\" (UID: \"e19f3dad-1f20-4cb5-8940-1408e9a22ae3\") " pod="kuadrant-system/authorino-6889cbd8c4-nhxns" Nov 03 19:58:26.232398 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.232398 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9mtc\" (UniqueName: \"kubernetes.io/projected/e19f3dad-1f20-4cb5-8940-1408e9a22ae3-kube-api-access-j9mtc\") pod \"authorino-6889cbd8c4-nhxns\" (UID: \"e19f3dad-1f20-4cb5-8940-1408e9a22ae3\") " pod="kuadrant-system/authorino-6889cbd8c4-nhxns" Nov 03 19:58:26.232696 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.232674 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglq9\" (UniqueName: \"kubernetes.io/projected/9682e516-2495-4ec2-8aaa-5e0514e0ce65-kube-api-access-hglq9\") pod \"authorino-6f6944bcdd-ms8vz\" (UID: \"9682e516-2495-4ec2-8aaa-5e0514e0ce65\") " pod="kuadrant-system/authorino-6f6944bcdd-ms8vz" Nov 03 19:58:26.241094 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.241041 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hglq9\" (UniqueName: \"kubernetes.io/projected/9682e516-2495-4ec2-8aaa-5e0514e0ce65-kube-api-access-hglq9\") pod \"authorino-6f6944bcdd-ms8vz\" (UID: \"9682e516-2495-4ec2-8aaa-5e0514e0ce65\") " pod="kuadrant-system/authorino-6f6944bcdd-ms8vz" Nov 03 19:58:26.333455 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.333414 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e19f3dad-1f20-4cb5-8940-1408e9a22ae3-tls-cert\") pod \"authorino-6889cbd8c4-nhxns\" (UID: \"e19f3dad-1f20-4cb5-8940-1408e9a22ae3\") " pod="kuadrant-system/authorino-6889cbd8c4-nhxns" Nov 03 19:58:26.333455 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.333457 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9mtc\" (UniqueName: \"kubernetes.io/projected/e19f3dad-1f20-4cb5-8940-1408e9a22ae3-kube-api-access-j9mtc\") pod \"authorino-6889cbd8c4-nhxns\" (UID: \"e19f3dad-1f20-4cb5-8940-1408e9a22ae3\") " pod="kuadrant-system/authorino-6889cbd8c4-nhxns" Nov 03 19:58:26.336040 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.336013 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e19f3dad-1f20-4cb5-8940-1408e9a22ae3-tls-cert\") pod \"authorino-6889cbd8c4-nhxns\" (UID: \"e19f3dad-1f20-4cb5-8940-1408e9a22ae3\") " pod="kuadrant-system/authorino-6889cbd8c4-nhxns" Nov 03 19:58:26.341459 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.341433 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9mtc\" (UniqueName: \"kubernetes.io/projected/e19f3dad-1f20-4cb5-8940-1408e9a22ae3-kube-api-access-j9mtc\") pod \"authorino-6889cbd8c4-nhxns\" (UID: \"e19f3dad-1f20-4cb5-8940-1408e9a22ae3\") " pod="kuadrant-system/authorino-6889cbd8c4-nhxns" Nov 03 19:58:26.414699 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.414663 5111 generic.go:356] "Generic (PLEG): container finished" podID="0f667fce-363f-476e-8601-afd78a7f3121" containerID="556de97197debf90d7ad5b1f2188cfa171a2e726d25c11fb54e706de9ecdb3b0" exitCode=0 Nov 03 19:58:26.414878 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.414736 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" event={"ID":"0f667fce-363f-476e-8601-afd78a7f3121","Type":"ContainerDied","Data":"556de97197debf90d7ad5b1f2188cfa171a2e726d25c11fb54e706de9ecdb3b0"} Nov 03 19:58:26.414878 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.414833 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ms8vz" Nov 03 19:58:26.415311 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.415291 5111 scope.go:122] "RemoveContainer" containerID="556de97197debf90d7ad5b1f2188cfa171a2e726d25c11fb54e706de9ecdb3b0" Nov 03 19:58:26.416589 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.416558 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:58:26.416651 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.416635 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:58:26.416690 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.416673 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:58:26.423999 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.423976 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ms8vz" Nov 03 19:58:26.434620 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.434581 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9682e516-2495-4ec2-8aaa-5e0514e0ce65-kube-api-access-hglq9\" (UniqueName: \"kubernetes.io/projected/9682e516-2495-4ec2-8aaa-5e0514e0ce65-kube-api-access-hglq9\") pod \"9682e516-2495-4ec2-8aaa-5e0514e0ce65\" (UID: \"9682e516-2495-4ec2-8aaa-5e0514e0ce65\") " Nov 03 19:58:26.436879 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.436856 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9682e516-2495-4ec2-8aaa-5e0514e0ce65-kube-api-access-hglq9" pod "9682e516-2495-4ec2-8aaa-5e0514e0ce65" (UID: "9682e516-2495-4ec2-8aaa-5e0514e0ce65"). InnerVolumeSpecName "kube-api-access-hglq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:26.441121 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.441099 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nhxns"] Nov 03 19:58:26.441350 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.441338 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-nhxns" Nov 03 19:58:26.465335 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.465302 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bbdff98d9-h9vgg"] Nov 03 19:58:26.491998 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.487290 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bbdff98d9-h9vgg"] Nov 03 19:58:26.491998 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.487410 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bbdff98d9-h9vgg" Nov 03 19:58:26.537286 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.536135 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b217d33f-76d9-4919-9161-1657d618930b-tls-cert\") pod \"authorino-bbdff98d9-h9vgg\" (UID: \"b217d33f-76d9-4919-9161-1657d618930b\") " pod="kuadrant-system/authorino-bbdff98d9-h9vgg" Nov 03 19:58:26.537286 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.536220 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqvqf\" (UniqueName: \"kubernetes.io/projected/b217d33f-76d9-4919-9161-1657d618930b-kube-api-access-dqvqf\") pod \"authorino-bbdff98d9-h9vgg\" (UID: \"b217d33f-76d9-4919-9161-1657d618930b\") " pod="kuadrant-system/authorino-bbdff98d9-h9vgg" Nov 03 19:58:26.537286 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.536282 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hglq9\" (UniqueName: \"kubernetes.io/projected/9682e516-2495-4ec2-8aaa-5e0514e0ce65-kube-api-access-hglq9\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:26.637173 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.637138 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b217d33f-76d9-4919-9161-1657d618930b-tls-cert\") pod \"authorino-bbdff98d9-h9vgg\" (UID: \"b217d33f-76d9-4919-9161-1657d618930b\") " pod="kuadrant-system/authorino-bbdff98d9-h9vgg" Nov 03 19:58:26.637340 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.637216 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvqf\" (UniqueName: \"kubernetes.io/projected/b217d33f-76d9-4919-9161-1657d618930b-kube-api-access-dqvqf\") pod \"authorino-bbdff98d9-h9vgg\" (UID: \"b217d33f-76d9-4919-9161-1657d618930b\") " pod="kuadrant-system/authorino-bbdff98d9-h9vgg" Nov 03 19:58:26.639686 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.639647 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b217d33f-76d9-4919-9161-1657d618930b-tls-cert\") pod \"authorino-bbdff98d9-h9vgg\" (UID: \"b217d33f-76d9-4919-9161-1657d618930b\") " pod="kuadrant-system/authorino-bbdff98d9-h9vgg" Nov 03 19:58:26.645229 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.645205 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqvqf\" (UniqueName: \"kubernetes.io/projected/b217d33f-76d9-4919-9161-1657d618930b-kube-api-access-dqvqf\") pod \"authorino-bbdff98d9-h9vgg\" (UID: \"b217d33f-76d9-4919-9161-1657d618930b\") " pod="kuadrant-system/authorino-bbdff98d9-h9vgg" Nov 03 19:58:26.746559 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.746516 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nhxns"] Nov 03 19:58:26.748571 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:58:26.748543 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode19f3dad_1f20_4cb5_8940_1408e9a22ae3.slice/crio-5ff25841771ddac961c77ecd2aad456f60d689edfec178be772606145672d0cf WatchSource:0}: Error finding container 5ff25841771ddac961c77ecd2aad456f60d689edfec178be772606145672d0cf: Status 404 returned error can't find the container with id 5ff25841771ddac961c77ecd2aad456f60d689edfec178be772606145672d0cf Nov 03 19:58:26.809896 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:26.807051 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bbdff98d9-h9vgg" Nov 03 19:58:27.104989 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:27.104953 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bbdff98d9-h9vgg"] Nov 03 19:58:27.108167 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:58:27.108141 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb217d33f_76d9_4919_9161_1657d618930b.slice/crio-7bba8164b250fd51890a70c5cd70b293ecae39d1067bf3885ff0ca37fef26a7b WatchSource:0}: Error finding container 7bba8164b250fd51890a70c5cd70b293ecae39d1067bf3885ff0ca37fef26a7b: Status 404 returned error can't find the container with id 7bba8164b250fd51890a70c5cd70b293ecae39d1067bf3885ff0ca37fef26a7b Nov 03 19:58:27.421807 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:27.421772 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" event={"ID":"0f667fce-363f-476e-8601-afd78a7f3121","Type":"ContainerStarted","Data":"fe9bef312215087c89b4e4f4e014ecc2813ca74b37f58c5b67a1d092cacf5aa4"} Nov 03 19:58:27.422978 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:27.422947 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bbdff98d9-h9vgg" event={"ID":"b217d33f-76d9-4919-9161-1657d618930b","Type":"ContainerStarted","Data":"7bba8164b250fd51890a70c5cd70b293ecae39d1067bf3885ff0ca37fef26a7b"} Nov 03 19:58:27.423989 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:27.423970 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ms8vz" Nov 03 19:58:27.423989 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:27.423981 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-nhxns" event={"ID":"e19f3dad-1f20-4cb5-8940-1408e9a22ae3","Type":"ContainerStarted","Data":"5ff25841771ddac961c77ecd2aad456f60d689edfec178be772606145672d0cf"} Nov 03 19:58:27.479124 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:27.479081 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ms8vz"] Nov 03 19:58:27.483217 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:27.483188 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ms8vz"] Nov 03 19:58:27.519358 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:27.519331 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:58:27.519617 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:27.519593 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-p994c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 19:58:27.519701 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:27.519647 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" podUID="0f667fce-363f-476e-8601-afd78a7f3121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 19:58:28.185568 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:28.185509 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9682e516-2495-4ec2-8aaa-5e0514e0ce65" path="/var/lib/kubelet/pods/9682e516-2495-4ec2-8aaa-5e0514e0ce65/volumes" Nov 03 19:58:28.438207 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:28.438082 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bbdff98d9-h9vgg" event={"ID":"b217d33f-76d9-4919-9161-1657d618930b","Type":"ContainerStarted","Data":"c5865edf5f68957a2602d3b62080d5f2529d25b24323b5754e2d411221f798e0"} Nov 03 19:58:28.439995 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:28.439961 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-nhxns" event={"ID":"e19f3dad-1f20-4cb5-8940-1408e9a22ae3","Type":"ContainerStarted","Data":"527ffa54e28231b202138609b760313a5ae109b5247769a64eb001be9fe96b6f"} Nov 03 19:58:28.440145 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:28.440007 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-nhxns" podUID="e19f3dad-1f20-4cb5-8940-1408e9a22ae3" containerName="authorino" containerID="cri-o://527ffa54e28231b202138609b760313a5ae109b5247769a64eb001be9fe96b6f" gracePeriod=30 Nov 03 19:58:28.489281 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:28.489236 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-bbdff98d9-h9vgg" podStartSLOduration=1.796493903 podStartE2EDuration="2.489222134s" totalImagesPullingTime="692.728231ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:28.457412946 +0000 UTC m=+1138.419791387" watchObservedRunningTime="2026-11-03 19:58:28.489222134 +0000 UTC m=+1138.451600590" Nov 03 19:58:28.490184 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:28.490137 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-nhxns" podStartSLOduration=1.544864655 podStartE2EDuration="2.490127395s" totalImagesPullingTime="945.26274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:28.48876679 +0000 UTC m=+1138.451145312" watchObservedRunningTime="2026-11-03 19:58:28.490127395 +0000 UTC m=+1138.452505835" Nov 03 19:58:28.520462 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:28.520424 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-p994c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 19:58:28.520658 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:28.520492 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" podUID="0f667fce-363f-476e-8601-afd78a7f3121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 19:58:28.777292 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:28.777262 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-nhxns" Nov 03 19:58:28.882172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:28.882138 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e19f3dad-1f20-4cb5-8940-1408e9a22ae3-tls-cert\" (UniqueName: \"kubernetes.io/secret/e19f3dad-1f20-4cb5-8940-1408e9a22ae3-tls-cert\") pod \"e19f3dad-1f20-4cb5-8940-1408e9a22ae3\" (UID: \"e19f3dad-1f20-4cb5-8940-1408e9a22ae3\") " Nov 03 19:58:28.882361 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:28.882248 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e19f3dad-1f20-4cb5-8940-1408e9a22ae3-kube-api-access-j9mtc\" (UniqueName: \"kubernetes.io/projected/e19f3dad-1f20-4cb5-8940-1408e9a22ae3-kube-api-access-j9mtc\") pod \"e19f3dad-1f20-4cb5-8940-1408e9a22ae3\" (UID: \"e19f3dad-1f20-4cb5-8940-1408e9a22ae3\") " Nov 03 19:58:28.884357 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:28.884334 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19f3dad-1f20-4cb5-8940-1408e9a22ae3-kube-api-access-j9mtc" pod "e19f3dad-1f20-4cb5-8940-1408e9a22ae3" (UID: "e19f3dad-1f20-4cb5-8940-1408e9a22ae3"). InnerVolumeSpecName "kube-api-access-j9mtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:28.892858 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:28.892834 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19f3dad-1f20-4cb5-8940-1408e9a22ae3-tls-cert" pod "e19f3dad-1f20-4cb5-8940-1408e9a22ae3" (UID: "e19f3dad-1f20-4cb5-8940-1408e9a22ae3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:28.982891 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:28.982859 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9mtc\" (UniqueName: \"kubernetes.io/projected/e19f3dad-1f20-4cb5-8940-1408e9a22ae3-kube-api-access-j9mtc\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:28.982891 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:28.982886 5111 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e19f3dad-1f20-4cb5-8940-1408e9a22ae3-tls-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:29.470902 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:29.470852 5111 generic.go:356] "Generic (PLEG): container finished" podID="e19f3dad-1f20-4cb5-8940-1408e9a22ae3" containerID="527ffa54e28231b202138609b760313a5ae109b5247769a64eb001be9fe96b6f" exitCode=0 Nov 03 19:58:29.470902 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:29.470891 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-nhxns" Nov 03 19:58:29.471490 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:29.470930 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-nhxns" event={"ID":"e19f3dad-1f20-4cb5-8940-1408e9a22ae3","Type":"ContainerDied","Data":"527ffa54e28231b202138609b760313a5ae109b5247769a64eb001be9fe96b6f"} Nov 03 19:58:29.471490 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:29.470973 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-nhxns" event={"ID":"e19f3dad-1f20-4cb5-8940-1408e9a22ae3","Type":"ContainerDied","Data":"5ff25841771ddac961c77ecd2aad456f60d689edfec178be772606145672d0cf"} Nov 03 19:58:29.471490 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:29.470991 5111 scope.go:122] "RemoveContainer" containerID="527ffa54e28231b202138609b760313a5ae109b5247769a64eb001be9fe96b6f" Nov 03 19:58:29.480172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:29.480153 5111 scope.go:122] "RemoveContainer" containerID="527ffa54e28231b202138609b760313a5ae109b5247769a64eb001be9fe96b6f" Nov 03 19:58:29.480432 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:58:29.480403 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527ffa54e28231b202138609b760313a5ae109b5247769a64eb001be9fe96b6f\": container with ID starting with 527ffa54e28231b202138609b760313a5ae109b5247769a64eb001be9fe96b6f not found: ID does not exist" containerID="527ffa54e28231b202138609b760313a5ae109b5247769a64eb001be9fe96b6f" Nov 03 19:58:29.480514 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:29.480443 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527ffa54e28231b202138609b760313a5ae109b5247769a64eb001be9fe96b6f"} err="failed to get container status \"527ffa54e28231b202138609b760313a5ae109b5247769a64eb001be9fe96b6f\": rpc error: code = NotFound desc = could not find container \"527ffa54e28231b202138609b760313a5ae109b5247769a64eb001be9fe96b6f\": container with ID starting with 527ffa54e28231b202138609b760313a5ae109b5247769a64eb001be9fe96b6f not found: ID does not exist" Nov 03 19:58:29.492366 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:29.492344 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nhxns"] Nov 03 19:58:29.495724 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:29.495706 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nhxns"] Nov 03 19:58:29.520224 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:29.520202 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-p994c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 19:58:29.520302 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:29.520245 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" podUID="0f667fce-363f-476e-8601-afd78a7f3121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 19:58:30.181035 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:30.181001 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e19f3dad-1f20-4cb5-8940-1408e9a22ae3" path="/var/lib/kubelet/pods/e19f3dad-1f20-4cb5-8940-1408e9a22ae3/volumes" Nov 03 19:58:30.520314 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:30.520229 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-p994c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 19:58:30.520314 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:30.520285 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" podUID="0f667fce-363f-476e-8601-afd78a7f3121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 19:58:31.519824 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:31.519791 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-p994c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 19:58:31.520014 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:31.519859 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" podUID="0f667fce-363f-476e-8601-afd78a7f3121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 19:58:32.523659 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:32.523629 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:58:32.524052 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:32.523923 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:58:32.524618 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:32.524603 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 19:58:35.055692 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:58:35.055651 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:58:45.045172 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.045136 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl"] Nov 03 19:58:45.045857 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.045568 5111 cpu_manager.go:414] "containerMap: removing container" podUID="e19f3dad-1f20-4cb5-8940-1408e9a22ae3" containerName="authorino" Nov 03 19:58:45.045857 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.045581 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e19f3dad-1f20-4cb5-8940-1408e9a22ae3" containerName="authorino" Nov 03 19:58:45.045857 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.045640 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="e19f3dad-1f20-4cb5-8940-1408e9a22ae3" containerName="authorino" Nov 03 19:58:45.050382 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.050355 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.070290 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.070263 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl"] Nov 03 19:58:45.119850 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.119814 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.119850 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.119854 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.120032 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.119931 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.120032 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.119978 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.120032 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.119999 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.120032 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.120014 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.120160 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.120032 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.120206 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.120182 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.120244 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.120225 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2zfb\" (UniqueName: \"kubernetes.io/projected/cf6540f3-53cd-4088-bf2e-e44670ad75b5-kube-api-access-g2zfb\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.221089 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.221050 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.221266 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.221098 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.221266 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.221150 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.221266 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.221185 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.221266 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.221216 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.221266 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.221240 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.221266 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.221260 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.221613 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.221342 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.221613 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.221384 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2zfb\" (UniqueName: \"kubernetes.io/projected/cf6540f3-53cd-4088-bf2e-e44670ad75b5-kube-api-access-g2zfb\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.221723 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.221652 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.221828 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.221800 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.221961 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.221879 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.221961 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.221929 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.222097 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.222068 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.223626 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.223606 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.223704 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.223693 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.228978 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.228953 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.229215 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.229194 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2zfb\" (UniqueName: \"kubernetes.io/projected/cf6540f3-53cd-4088-bf2e-e44670ad75b5-kube-api-access-g2zfb\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gmqdl\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.365062 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.364982 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:45.646403 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.646380 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl"] Nov 03 19:58:45.648835 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:58:45.648808 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6540f3_53cd_4088_bf2e_e44670ad75b5.slice/crio-92476df310503b1a5a2db2fff65d9f114c1d355c3829747ae2e7b64456988b32 WatchSource:0}: Error finding container 92476df310503b1a5a2db2fff65d9f114c1d355c3829747ae2e7b64456988b32: Status 404 returned error can't find the container with id 92476df310503b1a5a2db2fff65d9f114c1d355c3829747ae2e7b64456988b32 Nov 03 19:58:45.651060 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.651030 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:58:45.651129 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.651095 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:58:45.651129 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:45.651125 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:58:46.572647 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:46.572602 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" event={"ID":"cf6540f3-53cd-4088-bf2e-e44670ad75b5","Type":"ContainerStarted","Data":"17d12e28bc545d048b1502e48f9b99b07a62f71509428ee1a04a3e797e85bad9"} Nov 03 19:58:46.573087 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:46.572691 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" event={"ID":"cf6540f3-53cd-4088-bf2e-e44670ad75b5","Type":"ContainerStarted","Data":"92476df310503b1a5a2db2fff65d9f114c1d355c3829747ae2e7b64456988b32"} Nov 03 19:58:46.594364 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:46.594303 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" podStartSLOduration=1.59429113 podStartE2EDuration="1.59429113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:46.592041231 +0000 UTC m=+1156.554419670" watchObservedRunningTime="2026-11-03 19:58:46.59429113 +0000 UTC m=+1156.556669568" Nov 03 19:58:47.366555 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:47.365881 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:47.371002 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:47.370972 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gmqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:58:47.371149 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:47.371021 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" podUID="cf6540f3-53cd-4088-bf2e-e44670ad75b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 19:58:48.366569 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:48.365985 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gmqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:58:48.366569 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:48.366052 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" podUID="cf6540f3-53cd-4088-bf2e-e44670ad75b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 19:58:49.369996 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:49.369955 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gmqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:58:49.370436 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:49.370043 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" podUID="cf6540f3-53cd-4088-bf2e-e44670ad75b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 19:58:50.369972 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:50.369929 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gmqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:58:50.370182 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:50.370002 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" podUID="cf6540f3-53cd-4088-bf2e-e44670ad75b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 19:58:51.365993 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:51.365952 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gmqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 19:58:51.366353 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:51.366320 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" podUID="cf6540f3-53cd-4088-bf2e-e44670ad75b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 19:58:52.369072 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:52.369041 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:52.369454 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:52.369327 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:58:52.370128 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:58:52.370109 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 19:59:00.069099 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.069065 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl"] Nov 03 19:59:00.073145 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.073125 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.089128 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.089092 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl"] Nov 03 19:59:00.173818 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.173781 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d969be31-5822-40ef-be6f-c6627bfa5956-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.173818 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.173819 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d969be31-5822-40ef-be6f-c6627bfa5956-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.174015 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.173844 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d969be31-5822-40ef-be6f-c6627bfa5956-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.174015 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.173979 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d969be31-5822-40ef-be6f-c6627bfa5956-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.174015 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.174012 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgfvq\" (UniqueName: \"kubernetes.io/projected/d969be31-5822-40ef-be6f-c6627bfa5956-kube-api-access-xgfvq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.174138 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.174033 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d969be31-5822-40ef-be6f-c6627bfa5956-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.174138 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.174066 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d969be31-5822-40ef-be6f-c6627bfa5956-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.174254 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.174217 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d969be31-5822-40ef-be6f-c6627bfa5956-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.174309 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.174286 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d969be31-5822-40ef-be6f-c6627bfa5956-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.275131 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.275099 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d969be31-5822-40ef-be6f-c6627bfa5956-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.275312 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.275150 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d969be31-5822-40ef-be6f-c6627bfa5956-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.275312 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.275274 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d969be31-5822-40ef-be6f-c6627bfa5956-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.275441 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.275314 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d969be31-5822-40ef-be6f-c6627bfa5956-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.275441 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.275339 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d969be31-5822-40ef-be6f-c6627bfa5956-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.275441 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.275396 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d969be31-5822-40ef-be6f-c6627bfa5956-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.275441 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.275425 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgfvq\" (UniqueName: \"kubernetes.io/projected/d969be31-5822-40ef-be6f-c6627bfa5956-kube-api-access-xgfvq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.275654 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.275453 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d969be31-5822-40ef-be6f-c6627bfa5956-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.275654 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.275482 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d969be31-5822-40ef-be6f-c6627bfa5956-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.275654 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.275524 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d969be31-5822-40ef-be6f-c6627bfa5956-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.275778 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.275677 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d969be31-5822-40ef-be6f-c6627bfa5956-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.275985 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.275964 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d969be31-5822-40ef-be6f-c6627bfa5956-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.276061 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.276004 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d969be31-5822-40ef-be6f-c6627bfa5956-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.276061 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.276055 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d969be31-5822-40ef-be6f-c6627bfa5956-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.277505 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.277485 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d969be31-5822-40ef-be6f-c6627bfa5956-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.277771 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.277753 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d969be31-5822-40ef-be6f-c6627bfa5956-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.284068 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.284044 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgfvq\" (UniqueName: \"kubernetes.io/projected/d969be31-5822-40ef-be6f-c6627bfa5956-kube-api-access-xgfvq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.284068 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.284055 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d969be31-5822-40ef-be6f-c6627bfa5956-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h7vtl\" (UID: \"d969be31-5822-40ef-be6f-c6627bfa5956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.387750 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.387713 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:00.662889 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.662845 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl"] Nov 03 19:59:00.665208 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:59:00.665177 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd969be31_5822_40ef_be6f_c6627bfa5956.slice/crio-44968bf425d59410cbccedc21e07b5479d9f20dd0c3c1c15236f90c9de6bb90b WatchSource:0}: Error finding container 44968bf425d59410cbccedc21e07b5479d9f20dd0c3c1c15236f90c9de6bb90b: Status 404 returned error can't find the container with id 44968bf425d59410cbccedc21e07b5479d9f20dd0c3c1c15236f90c9de6bb90b Nov 03 19:59:00.667611 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.667578 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:59:00.667683 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.667650 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:59:00.667724 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:00.667682 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:59:01.643364 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:01.643322 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" event={"ID":"d969be31-5822-40ef-be6f-c6627bfa5956","Type":"ContainerStarted","Data":"df60dc4c43be56c554671893f48dbbd87787d80ca04f7790e61022106512be04"} Nov 03 19:59:01.643835 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:01.643393 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" event={"ID":"d969be31-5822-40ef-be6f-c6627bfa5956","Type":"ContainerStarted","Data":"44968bf425d59410cbccedc21e07b5479d9f20dd0c3c1c15236f90c9de6bb90b"} Nov 03 19:59:01.674006 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:01.673946 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" podStartSLOduration=1.673930296 podStartE2EDuration="1.673930296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:01.673001199 +0000 UTC m=+1171.635379640" watchObservedRunningTime="2026-11-03 19:59:01.673930296 +0000 UTC m=+1171.636308735" Nov 03 19:59:02.388508 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:02.388477 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:02.389772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:02.389747 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h7vtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 19:59:02.389886 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:02.389792 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" podUID="d969be31-5822-40ef-be6f-c6627bfa5956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 19:59:03.388914 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:03.388881 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h7vtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 19:59:03.389345 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:03.388934 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" podUID="d969be31-5822-40ef-be6f-c6627bfa5956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 19:59:04.389188 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:04.389147 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h7vtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 19:59:04.389822 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:04.389232 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" podUID="d969be31-5822-40ef-be6f-c6627bfa5956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 19:59:05.078608 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:59:05.078572 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:59:05.389057 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:05.389021 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h7vtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 19:59:05.389210 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:05.389080 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" podUID="d969be31-5822-40ef-be6f-c6627bfa5956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 19:59:06.389215 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:06.389167 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h7vtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 19:59:06.389674 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:06.389247 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" podUID="d969be31-5822-40ef-be6f-c6627bfa5956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 19:59:07.392214 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:07.392180 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:07.392628 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:07.392459 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:07.393083 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:07.393069 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" Nov 03 19:59:15.088054 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.088011 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx"] Nov 03 19:59:15.093250 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.093221 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.104211 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.104183 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.104468 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.104443 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.104695 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.104674 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.104848 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.104829 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.104970 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.104947 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g2v6\" (UniqueName: \"kubernetes.io/projected/cf5856f8-f452-4f05-b083-fb88f3db028a-kube-api-access-6g2v6\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.105108 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.104982 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.105108 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.105031 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.105108 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.105099 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.105233 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.105139 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf5856f8-f452-4f05-b083-fb88f3db028a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.106917 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.106891 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx"] Nov 03 19:59:15.206259 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.206220 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.206259 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.206261 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g2v6\" (UniqueName: \"kubernetes.io/projected/cf5856f8-f452-4f05-b083-fb88f3db028a-kube-api-access-6g2v6\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.206557 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.206280 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.206557 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.206412 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.206557 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.206510 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.206764 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.206580 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf5856f8-f452-4f05-b083-fb88f3db028a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.206764 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.206652 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.206764 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.206681 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.206764 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.206753 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.206982 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.206794 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.206982 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.206810 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.206982 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.206895 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.207224 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.207201 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.207309 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.207291 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf5856f8-f452-4f05-b083-fb88f3db028a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.209109 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.209070 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.209267 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.209248 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.214139 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.214114 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.214340 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.214325 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g2v6\" (UniqueName: \"kubernetes.io/projected/cf5856f8-f452-4f05-b083-fb88f3db028a-kube-api-access-6g2v6\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4vnbx\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.410574 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.410506 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:15.723692 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.723522 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx"] Nov 03 19:59:15.726654 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:59:15.726619 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5856f8_f452_4f05_b083_fb88f3db028a.slice/crio-b2745530e338da77b8b2b3d0b280d5ecfd8eeb3afa0db05df782c64e70ff7731 WatchSource:0}: Error finding container b2745530e338da77b8b2b3d0b280d5ecfd8eeb3afa0db05df782c64e70ff7731: Status 404 returned error can't find the container with id b2745530e338da77b8b2b3d0b280d5ecfd8eeb3afa0db05df782c64e70ff7731 Nov 03 19:59:15.729431 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.729398 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:59:15.729513 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.729469 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:59:15.729513 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:15.729498 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 19:59:16.736973 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:16.736940 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" event={"ID":"cf5856f8-f452-4f05-b083-fb88f3db028a","Type":"ContainerStarted","Data":"b34010d2adcefbfd7d4a1f28020eaf9d4c19f1f72b2a925cad89052bf186ef43"} Nov 03 19:59:16.737400 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:16.736992 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" event={"ID":"cf5856f8-f452-4f05-b083-fb88f3db028a","Type":"ContainerStarted","Data":"b2745530e338da77b8b2b3d0b280d5ecfd8eeb3afa0db05df782c64e70ff7731"} Nov 03 19:59:16.766552 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:16.762687 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" podStartSLOduration=1.762669034 podStartE2EDuration="1.762669034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:16.758451687 +0000 UTC m=+1186.720830127" watchObservedRunningTime="2026-11-03 19:59:16.762669034 +0000 UTC m=+1186.725047474" Nov 03 19:59:17.411001 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:17.410964 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:17.412496 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:17.412473 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4vnbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 19:59:17.412641 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:17.412518 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" podUID="cf5856f8-f452-4f05-b083-fb88f3db028a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 19:59:18.411188 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:18.411152 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4vnbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 19:59:18.411629 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:18.411215 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" podUID="cf5856f8-f452-4f05-b083-fb88f3db028a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 19:59:19.411040 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:19.411009 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4vnbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 19:59:19.411209 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:19.411069 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" podUID="cf5856f8-f452-4f05-b083-fb88f3db028a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 19:59:20.411370 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:20.411331 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4vnbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 19:59:20.411791 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:20.411389 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" podUID="cf5856f8-f452-4f05-b083-fb88f3db028a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 19:59:21.411232 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:21.411195 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4vnbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 19:59:21.411433 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:21.411256 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" podUID="cf5856f8-f452-4f05-b083-fb88f3db028a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 19:59:22.414686 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:22.414655 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:22.415094 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:22.414931 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:22.415630 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:22.415607 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 19:59:23.411757 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.411724 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-rvrqd"] Nov 03 19:59:23.415898 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.415873 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.420989 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.420960 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-rvrqd"] Nov 03 19:59:23.422564 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.422515 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2z695\"" Nov 03 19:59:23.422726 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.422645 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 19:59:23.426654 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.423145 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 19:59:23.426654 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.423203 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 19:59:23.510618 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.510585 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e49d0823-94c9-43a3-9963-72ed46e78ae8-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.510772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.510652 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e49d0823-94c9-43a3-9963-72ed46e78ae8-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.510772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.510722 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vngdr\" (UniqueName: \"kubernetes.io/projected/e49d0823-94c9-43a3-9963-72ed46e78ae8-kube-api-access-vngdr\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.510860 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.510781 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e49d0823-94c9-43a3-9963-72ed46e78ae8-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.510860 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.510801 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e49d0823-94c9-43a3-9963-72ed46e78ae8-home\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.510860 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.510816 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e49d0823-94c9-43a3-9963-72ed46e78ae8-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.612207 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.612169 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e49d0823-94c9-43a3-9963-72ed46e78ae8-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.612390 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.612214 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vngdr\" (UniqueName: \"kubernetes.io/projected/e49d0823-94c9-43a3-9963-72ed46e78ae8-kube-api-access-vngdr\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.612390 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.612256 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e49d0823-94c9-43a3-9963-72ed46e78ae8-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.612390 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.612275 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e49d0823-94c9-43a3-9963-72ed46e78ae8-home\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.612390 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.612289 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e49d0823-94c9-43a3-9963-72ed46e78ae8-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.612647 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.612395 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e49d0823-94c9-43a3-9963-72ed46e78ae8-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.612765 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.612743 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e49d0823-94c9-43a3-9963-72ed46e78ae8-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.612832 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.612797 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e49d0823-94c9-43a3-9963-72ed46e78ae8-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.612832 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.612822 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e49d0823-94c9-43a3-9963-72ed46e78ae8-home\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.614641 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.614621 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e49d0823-94c9-43a3-9963-72ed46e78ae8-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.614886 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.614866 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e49d0823-94c9-43a3-9963-72ed46e78ae8-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.620118 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.620096 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vngdr\" (UniqueName: \"kubernetes.io/projected/e49d0823-94c9-43a3-9963-72ed46e78ae8-kube-api-access-vngdr\") pod \"echo-simulator-kserve-64b6f8f96c-rvrqd\" (UID: \"e49d0823-94c9-43a3-9963-72ed46e78ae8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:23.733416 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:23.733328 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:24.009307 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:24.009281 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-rvrqd"] Nov 03 19:59:24.011659 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:59:24.011624 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49d0823_94c9_43a3_9963_72ed46e78ae8.slice/crio-819b3e9840772227bc914329f8fc7ba5ec02d011db8a0703244a6a269fb706df WatchSource:0}: Error finding container 819b3e9840772227bc914329f8fc7ba5ec02d011db8a0703244a6a269fb706df: Status 404 returned error can't find the container with id 819b3e9840772227bc914329f8fc7ba5ec02d011db8a0703244a6a269fb706df Nov 03 19:59:24.804520 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:24.804465 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" event={"ID":"e49d0823-94c9-43a3-9963-72ed46e78ae8","Type":"ContainerStarted","Data":"819b3e9840772227bc914329f8fc7ba5ec02d011db8a0703244a6a269fb706df"} Nov 03 19:59:26.836565 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:26.836059 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" event={"ID":"e49d0823-94c9-43a3-9963-72ed46e78ae8","Type":"ContainerStarted","Data":"8e65c44d967502d827709d6be10bc554603d880b049c9f04d2e9848e66ad8180"} Nov 03 19:59:26.873554 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:26.868291 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" podStartSLOduration=1.757681684 podStartE2EDuration="3.868271864s" totalImagesPullingTime="2.11059018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:26.865926483 +0000 UTC m=+1196.828304924" watchObservedRunningTime="2026-11-03 19:59:26.868271864 +0000 UTC m=+1196.830650304" Nov 03 19:59:27.733772 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:27.733735 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:28.273484 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.273447 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r"] Nov 03 19:59:28.277946 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.277928 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.287647 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.287622 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r"] Nov 03 19:59:28.288642 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.288618 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 19:59:28.406319 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.403147 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.406319 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.403227 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.406319 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.403261 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.406319 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.403299 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.406319 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.403371 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.406319 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.403458 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jqp5\" (UniqueName: \"kubernetes.io/projected/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-kube-api-access-8jqp5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.504226 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.504191 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.504226 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.504226 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.504482 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.504255 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.504482 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.504333 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.504482 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.504412 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jqp5\" (UniqueName: \"kubernetes.io/projected/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-kube-api-access-8jqp5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.504482 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.504460 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.504784 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.504729 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.504911 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.504800 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.504911 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.504800 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.506790 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.506768 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.506925 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.506908 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.512229 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.512208 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jqp5\" (UniqueName: \"kubernetes.io/projected/e52a6c08-4959-4cf8-90a6-fe48e0dbd12b-kube-api-access-8jqp5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r\" (UID: \"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.598188 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.595726 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:28.890804 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:28.890770 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r"] Nov 03 19:59:29.935558 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:29.934862 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" event={"ID":"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b","Type":"ContainerStarted","Data":"86d139ecdb6e7999fcfd0406eac879c86c3ca960c865800a20b4632c9df62e79"} Nov 03 19:59:30.349298 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:30.349225 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 19:59:30.349742 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:30.349727 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 19:59:30.358939 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:30.358919 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 19:59:30.360352 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:30.360327 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 19:59:32.703134 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.703084 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh"] Nov 03 19:59:32.708164 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.708129 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh"] Nov 03 19:59:32.708456 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.708425 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.714174 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.714119 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 19:59:32.838049 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.838008 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/457e9dde-adec-4ec7-83bd-8d157b9a3362-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.838450 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.838391 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/457e9dde-adec-4ec7-83bd-8d157b9a3362-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.838591 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.838473 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkpds\" (UniqueName: \"kubernetes.io/projected/457e9dde-adec-4ec7-83bd-8d157b9a3362-kube-api-access-fkpds\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.838670 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.838610 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/457e9dde-adec-4ec7-83bd-8d157b9a3362-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.838670 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.838655 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/457e9dde-adec-4ec7-83bd-8d157b9a3362-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.838819 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.838801 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457e9dde-adec-4ec7-83bd-8d157b9a3362-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.940544 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.940487 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/457e9dde-adec-4ec7-83bd-8d157b9a3362-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.940885 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.940862 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/457e9dde-adec-4ec7-83bd-8d157b9a3362-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.941048 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.941031 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkpds\" (UniqueName: \"kubernetes.io/projected/457e9dde-adec-4ec7-83bd-8d157b9a3362-kube-api-access-fkpds\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.941613 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.941587 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/457e9dde-adec-4ec7-83bd-8d157b9a3362-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.942127 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.942098 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/457e9dde-adec-4ec7-83bd-8d157b9a3362-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.942704 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.942671 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457e9dde-adec-4ec7-83bd-8d157b9a3362-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.943288 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.942037 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/457e9dde-adec-4ec7-83bd-8d157b9a3362-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.943417 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.940942 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/457e9dde-adec-4ec7-83bd-8d157b9a3362-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.943519 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.941194 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/457e9dde-adec-4ec7-83bd-8d157b9a3362-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.946116 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.945127 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/457e9dde-adec-4ec7-83bd-8d157b9a3362-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.948979 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.948934 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457e9dde-adec-4ec7-83bd-8d157b9a3362-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.949840 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.949815 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkpds\" (UniqueName: \"kubernetes.io/projected/457e9dde-adec-4ec7-83bd-8d157b9a3362-kube-api-access-fkpds\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8x8xh\" (UID: \"457e9dde-adec-4ec7-83bd-8d157b9a3362\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:32.971604 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.969235 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" event={"ID":"e52a6c08-4959-4cf8-90a6-fe48e0dbd12b","Type":"ContainerStarted","Data":"bb47fcbf7e857eced4a4b18be82116eba586529cd2e9dd411424a014ab2e47ba"} Nov 03 19:59:32.994555 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:32.991679 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podStartSLOduration=1.945005222 podStartE2EDuration="4.991658195s" totalImagesPullingTime="3.046652973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:32.990117718 +0000 UTC m=+1202.952496159" watchObservedRunningTime="2026-11-03 19:59:32.991658195 +0000 UTC m=+1202.954036636" Nov 03 19:59:33.036557 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:33.035749 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:33.644887 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:33.644856 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:33.698556 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:33.694883 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh"] Nov 03 19:59:33.740355 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:33.740324 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm"] Nov 03 19:59:33.900596 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:33.897946 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm"] Nov 03 19:59:33.923562 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:33.920674 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:33.923562 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:33.920839 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:33.923562 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:33.920931 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.007713 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.006048 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 19:59:34.023554 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.021655 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/989fbc44-448a-4608-904f-b265dcaccce5-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.023554 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.021760 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/989fbc44-448a-4608-904f-b265dcaccce5-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.023554 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.021794 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrhcl\" (UniqueName: \"kubernetes.io/projected/989fbc44-448a-4608-904f-b265dcaccce5-kube-api-access-wrhcl\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.023554 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.021825 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/989fbc44-448a-4608-904f-b265dcaccce5-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.023554 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.021879 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/989fbc44-448a-4608-904f-b265dcaccce5-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.023554 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.021911 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/989fbc44-448a-4608-904f-b265dcaccce5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.059554 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.054236 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" Nov 03 19:59:34.146559 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.141958 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/989fbc44-448a-4608-904f-b265dcaccce5-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.146559 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.142021 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrhcl\" (UniqueName: \"kubernetes.io/projected/989fbc44-448a-4608-904f-b265dcaccce5-kube-api-access-wrhcl\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.146559 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.142061 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/989fbc44-448a-4608-904f-b265dcaccce5-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.146559 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.142140 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/989fbc44-448a-4608-904f-b265dcaccce5-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.146559 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.142181 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/989fbc44-448a-4608-904f-b265dcaccce5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.146559 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.142268 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/989fbc44-448a-4608-904f-b265dcaccce5-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.146559 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.142932 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/989fbc44-448a-4608-904f-b265dcaccce5-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.146559 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.144388 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/989fbc44-448a-4608-904f-b265dcaccce5-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.171598 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.166561 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrhcl\" (UniqueName: \"kubernetes.io/projected/989fbc44-448a-4608-904f-b265dcaccce5-kube-api-access-wrhcl\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.171598 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.170365 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/989fbc44-448a-4608-904f-b265dcaccce5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.204555 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.201576 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/989fbc44-448a-4608-904f-b265dcaccce5-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.204555 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.203287 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/989fbc44-448a-4608-904f-b265dcaccce5-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c8tdm\" (UID: \"989fbc44-448a-4608-904f-b265dcaccce5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.300555 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.299605 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" event={"ID":"457e9dde-adec-4ec7-83bd-8d157b9a3362","Type":"ContainerStarted","Data":"74dd4be2afe0a1973b55afbb3e9ddd755b19ab7c8b87f8522e741cad160c9eab"} Nov 03 19:59:34.451324 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.451270 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:34.770067 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:34.769976 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm"] Nov 03 19:59:34.782344 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:59:34.782307 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989fbc44_448a_4608_904f_b265dcaccce5.slice/crio-67867cf52fed8cfa41a2be67fde715204a8ffd6a563b89396352cb45a1a1bd64 WatchSource:0}: Error finding container 67867cf52fed8cfa41a2be67fde715204a8ffd6a563b89396352cb45a1a1bd64: Status 404 returned error can't find the container with id 67867cf52fed8cfa41a2be67fde715204a8ffd6a563b89396352cb45a1a1bd64 Nov 03 19:59:35.145567 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:59:35.144260 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 19:59:35.332557 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:35.332078 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" event={"ID":"457e9dde-adec-4ec7-83bd-8d157b9a3362","Type":"ContainerStarted","Data":"44ffdf45d99bca9be639bb9325ab6770300471cc86866d25529cc16d76cbfdbd"} Nov 03 19:59:35.356556 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:35.353479 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" event={"ID":"989fbc44-448a-4608-904f-b265dcaccce5","Type":"ContainerStarted","Data":"67867cf52fed8cfa41a2be67fde715204a8ffd6a563b89396352cb45a1a1bd64"} Nov 03 19:59:36.036673 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.036618 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:36.373247 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.373215 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" event={"ID":"989fbc44-448a-4608-904f-b265dcaccce5","Type":"ContainerStarted","Data":"3c80acd888e844eb7e975073315c5afe83bb2ef53f243789e85b2d84389112cd"} Nov 03 19:59:36.396577 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.396489 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" podStartSLOduration=3.771558867 podStartE2EDuration="4.396469532s" totalImagesPullingTime="624.910665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:35.371901755 +0000 UTC m=+1205.334280196" watchObservedRunningTime="2026-11-03 19:59:36.396469532 +0000 UTC m=+1206.358847970" Nov 03 19:59:36.453459 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.453424 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:36.748834 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.748713 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" podStartSLOduration=3.139006377 podStartE2EDuration="3.748692755s" totalImagesPullingTime="609.686378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:36.39880327 +0000 UTC m=+1206.361181716" watchObservedRunningTime="2026-11-03 19:59:36.748692755 +0000 UTC m=+1206.711071196" Nov 03 19:59:36.752563 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.752519 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6"] Nov 03 19:59:36.756341 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.756313 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.766283 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.766258 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 19:59:36.768281 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.768262 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6"] Nov 03 19:59:36.813459 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.813427 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e292de6-b36f-481b-9d69-1961e7d8026b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.813459 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.813462 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsw4c\" (UniqueName: \"kubernetes.io/projected/1e292de6-b36f-481b-9d69-1961e7d8026b-kube-api-access-vsw4c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.813679 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.813489 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e292de6-b36f-481b-9d69-1961e7d8026b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.813716 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.813698 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e292de6-b36f-481b-9d69-1961e7d8026b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.813795 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.813747 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e292de6-b36f-481b-9d69-1961e7d8026b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.813905 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.813848 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e292de6-b36f-481b-9d69-1961e7d8026b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.916501 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.915714 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e292de6-b36f-481b-9d69-1961e7d8026b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.916501 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.915772 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e292de6-b36f-481b-9d69-1961e7d8026b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.916501 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.915813 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e292de6-b36f-481b-9d69-1961e7d8026b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.916501 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.915930 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e292de6-b36f-481b-9d69-1961e7d8026b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.916501 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.915986 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsw4c\" (UniqueName: \"kubernetes.io/projected/1e292de6-b36f-481b-9d69-1961e7d8026b-kube-api-access-vsw4c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.916501 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.916025 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e292de6-b36f-481b-9d69-1961e7d8026b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.916978 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.916844 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e292de6-b36f-481b-9d69-1961e7d8026b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.919393 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.918849 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e292de6-b36f-481b-9d69-1961e7d8026b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.919393 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.919104 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e292de6-b36f-481b-9d69-1961e7d8026b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.930399 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.930325 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsw4c\" (UniqueName: \"kubernetes.io/projected/1e292de6-b36f-481b-9d69-1961e7d8026b-kube-api-access-vsw4c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.931849 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.931826 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e292de6-b36f-481b-9d69-1961e7d8026b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:36.939329 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:36.939304 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e292de6-b36f-481b-9d69-1961e7d8026b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6\" (UID: \"1e292de6-b36f-481b-9d69-1961e7d8026b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:37.074165 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:37.074075 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:37.421555 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:37.421499 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6"] Nov 03 19:59:37.440801 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:59:37.435288 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e292de6_b36f_481b_9d69_1961e7d8026b.slice/crio-cd83c33a1d400047c8e0e3d9152a17e53e48f449e660dd0532fe20a718f63d73 WatchSource:0}: Error finding container cd83c33a1d400047c8e0e3d9152a17e53e48f449e660dd0532fe20a718f63d73: Status 404 returned error can't find the container with id cd83c33a1d400047c8e0e3d9152a17e53e48f449e660dd0532fe20a718f63d73 Nov 03 19:59:38.430202 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:38.430059 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" event={"ID":"1e292de6-b36f-481b-9d69-1961e7d8026b","Type":"ContainerStarted","Data":"b213add754145b1e64e4717db2a8ddbf2633831844e5e5f508884a12f375a6e9"} Nov 03 19:59:38.430202 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:38.430142 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" event={"ID":"1e292de6-b36f-481b-9d69-1961e7d8026b","Type":"ContainerStarted","Data":"cd83c33a1d400047c8e0e3d9152a17e53e48f449e660dd0532fe20a718f63d73"} Nov 03 19:59:38.616742 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:38.616707 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:39.047428 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:39.047396 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:39.074912 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:39.074882 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:39.077174 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:39.077119 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" podStartSLOduration=2.624679118 podStartE2EDuration="3.077098596s" totalImagesPullingTime="452.419478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:38.450099177 +0000 UTC m=+1208.412477617" watchObservedRunningTime="2026-11-03 19:59:39.077098596 +0000 UTC m=+1209.039477038" Nov 03 19:59:39.587861 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:39.587831 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 19:59:43.036567 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:43.036460 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:43.049048 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:43.049023 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:43.515358 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:43.515330 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" Nov 03 19:59:44.452897 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:44.452862 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:44.465255 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:44.465234 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:44.520968 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:44.520944 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" Nov 03 19:59:47.074870 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:47.074836 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:47.087125 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:47.087098 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:47.536794 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:47.536764 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" Nov 03 19:59:49.520711 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:49.520670 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77c9c58d6b-22kjj"] Nov 03 19:59:49.526656 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:49.526622 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77c9c58d6b-22kjj" Nov 03 19:59:49.532215 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:49.532186 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77c9c58d6b-22kjj"] Nov 03 19:59:49.666401 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:49.666368 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99039821-d164-41f5-b9c3-85fd95744441-tls-cert\") pod \"authorino-77c9c58d6b-22kjj\" (UID: \"99039821-d164-41f5-b9c3-85fd95744441\") " pod="kuadrant-system/authorino-77c9c58d6b-22kjj" Nov 03 19:59:49.666614 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:49.666484 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4mr9\" (UniqueName: \"kubernetes.io/projected/99039821-d164-41f5-b9c3-85fd95744441-kube-api-access-h4mr9\") pod \"authorino-77c9c58d6b-22kjj\" (UID: \"99039821-d164-41f5-b9c3-85fd95744441\") " pod="kuadrant-system/authorino-77c9c58d6b-22kjj" Nov 03 19:59:49.767286 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:49.767245 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99039821-d164-41f5-b9c3-85fd95744441-tls-cert\") pod \"authorino-77c9c58d6b-22kjj\" (UID: \"99039821-d164-41f5-b9c3-85fd95744441\") " pod="kuadrant-system/authorino-77c9c58d6b-22kjj" Nov 03 19:59:49.767481 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:49.767325 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4mr9\" (UniqueName: \"kubernetes.io/projected/99039821-d164-41f5-b9c3-85fd95744441-kube-api-access-h4mr9\") pod \"authorino-77c9c58d6b-22kjj\" (UID: \"99039821-d164-41f5-b9c3-85fd95744441\") " pod="kuadrant-system/authorino-77c9c58d6b-22kjj" Nov 03 19:59:49.769880 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:49.769854 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99039821-d164-41f5-b9c3-85fd95744441-tls-cert\") pod \"authorino-77c9c58d6b-22kjj\" (UID: \"99039821-d164-41f5-b9c3-85fd95744441\") " pod="kuadrant-system/authorino-77c9c58d6b-22kjj" Nov 03 19:59:49.775550 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:49.775479 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4mr9\" (UniqueName: \"kubernetes.io/projected/99039821-d164-41f5-b9c3-85fd95744441-kube-api-access-h4mr9\") pod \"authorino-77c9c58d6b-22kjj\" (UID: \"99039821-d164-41f5-b9c3-85fd95744441\") " pod="kuadrant-system/authorino-77c9c58d6b-22kjj" Nov 03 19:59:49.849476 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:49.849441 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77c9c58d6b-22kjj" Nov 03 19:59:50.122180 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:50.122153 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77c9c58d6b-22kjj"] Nov 03 19:59:50.124828 ip-10-0-134-178 kubenswrapper[5111]: W1103 19:59:50.124804 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99039821_d164_41f5_b9c3_85fd95744441.slice/crio-ec67c6f19e5dde1f527bca8fe57c606f5f278db5ec5bdc5b283fcb44b864bfab WatchSource:0}: Error finding container ec67c6f19e5dde1f527bca8fe57c606f5f278db5ec5bdc5b283fcb44b864bfab: Status 404 returned error can't find the container with id ec67c6f19e5dde1f527bca8fe57c606f5f278db5ec5bdc5b283fcb44b864bfab Nov 03 19:59:50.552070 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:50.551986 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77c9c58d6b-22kjj" event={"ID":"99039821-d164-41f5-b9c3-85fd95744441","Type":"ContainerStarted","Data":"ec67c6f19e5dde1f527bca8fe57c606f5f278db5ec5bdc5b283fcb44b864bfab"} Nov 03 19:59:51.563067 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:51.563032 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77c9c58d6b-22kjj" event={"ID":"99039821-d164-41f5-b9c3-85fd95744441","Type":"ContainerStarted","Data":"9a2b0a72b8c7e96dd8d8b2d07115a640e5248045466c6726e2c64ddb3e92520f"} Nov 03 19:59:51.581957 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:51.581902 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-77c9c58d6b-22kjj" podStartSLOduration=1.8203715759999999 podStartE2EDuration="2.581884715s" totalImagesPullingTime="761.513139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:51.580636515 +0000 UTC m=+1221.543014956" watchObservedRunningTime="2026-11-03 19:59:51.581884715 +0000 UTC m=+1221.544263154" Nov 03 19:59:51.617666 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:51.617604 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bbdff98d9-h9vgg"] Nov 03 19:59:51.617925 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:51.617898 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-bbdff98d9-h9vgg" podUID="b217d33f-76d9-4919-9161-1657d618930b" containerName="authorino" containerID="cri-o://c5865edf5f68957a2602d3b62080d5f2529d25b24323b5754e2d411221f798e0" gracePeriod=30 Nov 03 19:59:51.881725 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:51.881698 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bbdff98d9-h9vgg" Nov 03 19:59:51.899045 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:51.899019 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b217d33f-76d9-4919-9161-1657d618930b-kube-api-access-dqvqf\" (UniqueName: \"kubernetes.io/projected/b217d33f-76d9-4919-9161-1657d618930b-kube-api-access-dqvqf\") pod \"b217d33f-76d9-4919-9161-1657d618930b\" (UID: \"b217d33f-76d9-4919-9161-1657d618930b\") " Nov 03 19:59:51.899197 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:51.899091 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b217d33f-76d9-4919-9161-1657d618930b-tls-cert\" (UniqueName: \"kubernetes.io/secret/b217d33f-76d9-4919-9161-1657d618930b-tls-cert\") pod \"b217d33f-76d9-4919-9161-1657d618930b\" (UID: \"b217d33f-76d9-4919-9161-1657d618930b\") " Nov 03 19:59:51.901909 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:51.901881 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b217d33f-76d9-4919-9161-1657d618930b-kube-api-access-dqvqf" pod "b217d33f-76d9-4919-9161-1657d618930b" (UID: "b217d33f-76d9-4919-9161-1657d618930b"). InnerVolumeSpecName "kube-api-access-dqvqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:59:51.920375 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:51.920346 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b217d33f-76d9-4919-9161-1657d618930b-tls-cert" pod "b217d33f-76d9-4919-9161-1657d618930b" (UID: "b217d33f-76d9-4919-9161-1657d618930b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:59:52.000652 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:52.000608 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqvqf\" (UniqueName: \"kubernetes.io/projected/b217d33f-76d9-4919-9161-1657d618930b-kube-api-access-dqvqf\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:52.000652 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:52.000647 5111 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b217d33f-76d9-4919-9161-1657d618930b-tls-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:52.569598 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:52.569493 5111 generic.go:356] "Generic (PLEG): container finished" podID="b217d33f-76d9-4919-9161-1657d618930b" containerID="c5865edf5f68957a2602d3b62080d5f2529d25b24323b5754e2d411221f798e0" exitCode=0 Nov 03 19:59:52.569598 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:52.569566 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bbdff98d9-h9vgg" Nov 03 19:59:52.570193 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:52.569565 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bbdff98d9-h9vgg" event={"ID":"b217d33f-76d9-4919-9161-1657d618930b","Type":"ContainerDied","Data":"c5865edf5f68957a2602d3b62080d5f2529d25b24323b5754e2d411221f798e0"} Nov 03 19:59:52.570193 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:52.569683 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bbdff98d9-h9vgg" event={"ID":"b217d33f-76d9-4919-9161-1657d618930b","Type":"ContainerDied","Data":"7bba8164b250fd51890a70c5cd70b293ecae39d1067bf3885ff0ca37fef26a7b"} Nov 03 19:59:52.570193 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:52.569701 5111 scope.go:122] "RemoveContainer" containerID="c5865edf5f68957a2602d3b62080d5f2529d25b24323b5754e2d411221f798e0" Nov 03 19:59:52.580148 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:52.580124 5111 scope.go:122] "RemoveContainer" containerID="c5865edf5f68957a2602d3b62080d5f2529d25b24323b5754e2d411221f798e0" Nov 03 19:59:52.580413 ip-10-0-134-178 kubenswrapper[5111]: E1103 19:59:52.580395 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5865edf5f68957a2602d3b62080d5f2529d25b24323b5754e2d411221f798e0\": container with ID starting with c5865edf5f68957a2602d3b62080d5f2529d25b24323b5754e2d411221f798e0 not found: ID does not exist" containerID="c5865edf5f68957a2602d3b62080d5f2529d25b24323b5754e2d411221f798e0" Nov 03 19:59:52.580486 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:52.580421 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5865edf5f68957a2602d3b62080d5f2529d25b24323b5754e2d411221f798e0"} err="failed to get container status \"c5865edf5f68957a2602d3b62080d5f2529d25b24323b5754e2d411221f798e0\": rpc error: code = NotFound desc = could not find container \"c5865edf5f68957a2602d3b62080d5f2529d25b24323b5754e2d411221f798e0\": container with ID starting with c5865edf5f68957a2602d3b62080d5f2529d25b24323b5754e2d411221f798e0 not found: ID does not exist" Nov 03 19:59:52.587352 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:52.587331 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bbdff98d9-h9vgg"] Nov 03 19:59:52.591980 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:52.591956 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-bbdff98d9-h9vgg"] Nov 03 19:59:54.181725 ip-10-0-134-178 kubenswrapper[5111]: I1103 19:59:54.181692 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b217d33f-76d9-4919-9161-1657d618930b" path="/var/lib/kubelet/pods/b217d33f-76d9-4919-9161-1657d618930b/volumes" Nov 03 20:00:00.135793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.135758 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp"] Nov 03 20:00:00.136232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.136217 5111 cpu_manager.go:414] "containerMap: removing container" podUID="b217d33f-76d9-4919-9161-1657d618930b" containerName="authorino" Nov 03 20:00:00.136278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.136233 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b217d33f-76d9-4919-9161-1657d618930b" containerName="authorino" Nov 03 20:00:00.136359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.136349 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="b217d33f-76d9-4919-9161-1657d618930b" containerName="authorino" Nov 03 20:00:00.144962 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.144931 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" Nov 03 20:00:00.145934 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.145906 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp"] Nov 03 20:00:00.148618 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.148592 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:00:00.151065 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.151043 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:00:00.151314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.151267 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:00:00.194858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.194829 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gcvm\" (UniqueName: \"kubernetes.io/projected/6c233789-9149-4c70-871d-8f3b1fe188e3-kube-api-access-7gcvm\") pod \"maas-api-key-cleanup-29895600-cxfdp\" (UID: \"6c233789-9149-4c70-871d-8f3b1fe188e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" Nov 03 20:00:00.195015 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.194869 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c233789-9149-4c70-871d-8f3b1fe188e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895600-cxfdp\" (UID: \"6c233789-9149-4c70-871d-8f3b1fe188e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" Nov 03 20:00:00.296114 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.296077 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gcvm\" (UniqueName: \"kubernetes.io/projected/6c233789-9149-4c70-871d-8f3b1fe188e3-kube-api-access-7gcvm\") pod \"maas-api-key-cleanup-29895600-cxfdp\" (UID: \"6c233789-9149-4c70-871d-8f3b1fe188e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" Nov 03 20:00:00.296283 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.296128 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c233789-9149-4c70-871d-8f3b1fe188e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895600-cxfdp\" (UID: \"6c233789-9149-4c70-871d-8f3b1fe188e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" Nov 03 20:00:00.296848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.296828 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c233789-9149-4c70-871d-8f3b1fe188e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895600-cxfdp\" (UID: \"6c233789-9149-4c70-871d-8f3b1fe188e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" Nov 03 20:00:00.304706 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.304685 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gcvm\" (UniqueName: \"kubernetes.io/projected/6c233789-9149-4c70-871d-8f3b1fe188e3-kube-api-access-7gcvm\") pod \"maas-api-key-cleanup-29895600-cxfdp\" (UID: \"6c233789-9149-4c70-871d-8f3b1fe188e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" Nov 03 20:00:00.461800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.461709 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" Nov 03 20:00:00.737701 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:00.737660 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp"] Nov 03 20:00:00.744378 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:00:00.744348 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c233789_9149_4c70_871d_8f3b1fe188e3.slice/crio-f2ee641c3f73200d0a5f023d3862abb85b8353863fb3e6dcd17724479aa143b8 WatchSource:0}: Error finding container f2ee641c3f73200d0a5f023d3862abb85b8353863fb3e6dcd17724479aa143b8: Status 404 returned error can't find the container with id f2ee641c3f73200d0a5f023d3862abb85b8353863fb3e6dcd17724479aa143b8 Nov 03 20:00:01.616715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:01.616681 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" event={"ID":"6c233789-9149-4c70-871d-8f3b1fe188e3","Type":"ContainerStarted","Data":"f2ee641c3f73200d0a5f023d3862abb85b8353863fb3e6dcd17724479aa143b8"} Nov 03 20:00:04.634432 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:04.634392 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" event={"ID":"6c233789-9149-4c70-871d-8f3b1fe188e3","Type":"ContainerStarted","Data":"116642a5e363c5aa6d758d37b864f0eaa6768632009df1ba71762f667665257a"} Nov 03 20:00:04.653049 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:04.652999 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" podStartSLOduration=1.279586614 podStartE2EDuration="4.652986262s" totalImagesPullingTime="3.373399648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:04.650580319 +0000 UTC m=+1234.612958756" watchObservedRunningTime="2026-11-03 20:00:04.652986262 +0000 UTC m=+1234.615364746" Nov 03 20:00:05.055179 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:00:05.055141 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:00:05.641358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:05.641325 5111 generic.go:356] "Generic (PLEG): container finished" podID="6c233789-9149-4c70-871d-8f3b1fe188e3" containerID="116642a5e363c5aa6d758d37b864f0eaa6768632009df1ba71762f667665257a" exitCode=60 Nov 03 20:00:05.641856 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:05.641389 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" event={"ID":"6c233789-9149-4c70-871d-8f3b1fe188e3","Type":"ContainerDied","Data":"116642a5e363c5aa6d758d37b864f0eaa6768632009df1ba71762f667665257a"} Nov 03 20:00:05.641856 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:05.641703 5111 scope.go:122] "RemoveContainer" containerID="116642a5e363c5aa6d758d37b864f0eaa6768632009df1ba71762f667665257a" Nov 03 20:00:06.648228 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:06.648196 5111 generic.go:356] "Generic (PLEG): container finished" podID="6c233789-9149-4c70-871d-8f3b1fe188e3" containerID="4a5e974b5a3b4f645cb0cfb8ba9c47d07ed86b871ae08966c8811e0afd768859" exitCode=60 Nov 03 20:00:06.648623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:06.648263 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" event={"ID":"6c233789-9149-4c70-871d-8f3b1fe188e3","Type":"ContainerDied","Data":"4a5e974b5a3b4f645cb0cfb8ba9c47d07ed86b871ae08966c8811e0afd768859"} Nov 03 20:00:06.648623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:06.648331 5111 scope.go:122] "RemoveContainer" containerID="116642a5e363c5aa6d758d37b864f0eaa6768632009df1ba71762f667665257a" Nov 03 20:00:06.648623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:06.648568 5111 scope.go:122] "RemoveContainer" containerID="4a5e974b5a3b4f645cb0cfb8ba9c47d07ed86b871ae08966c8811e0afd768859" Nov 03 20:00:06.648866 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:00:06.648846 5111 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895600-cxfdp_odh-ai-gateway-infra(6c233789-9149-4c70-871d-8f3b1fe188e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" podUID="6c233789-9149-4c70-871d-8f3b1fe188e3" Nov 03 20:00:07.654891 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:07.654862 5111 scope.go:122] "RemoveContainer" containerID="4a5e974b5a3b4f645cb0cfb8ba9c47d07ed86b871ae08966c8811e0afd768859" Nov 03 20:00:07.655300 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:00:07.655094 5111 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895600-cxfdp_odh-ai-gateway-infra(6c233789-9149-4c70-871d-8f3b1fe188e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" podUID="6c233789-9149-4c70-871d-8f3b1fe188e3" Nov 03 20:00:08.459347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:08.459310 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-656bfc8bbd-llxlr"] Nov 03 20:00:08.464288 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:08.464268 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-656bfc8bbd-llxlr" Nov 03 20:00:08.468743 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:08.468717 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-wcnrg\"" Nov 03 20:00:08.468839 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:08.468804 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 20:00:08.468907 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:08.468830 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 20:00:08.470920 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:08.470898 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-656bfc8bbd-llxlr"] Nov 03 20:00:08.578397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:08.578364 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttsrw\" (UniqueName: \"kubernetes.io/projected/2716609d-ca4a-4bdf-8d24-3f06060a01e4-kube-api-access-ttsrw\") pod \"ai-gateway-controller-656bfc8bbd-llxlr\" (UID: \"2716609d-ca4a-4bdf-8d24-3f06060a01e4\") " pod="opendatahub/ai-gateway-controller-656bfc8bbd-llxlr" Nov 03 20:00:08.578579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:08.578518 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2716609d-ca4a-4bdf-8d24-3f06060a01e4-metrics-certs\") pod \"ai-gateway-controller-656bfc8bbd-llxlr\" (UID: \"2716609d-ca4a-4bdf-8d24-3f06060a01e4\") " pod="opendatahub/ai-gateway-controller-656bfc8bbd-llxlr" Nov 03 20:00:08.679422 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:08.679385 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttsrw\" (UniqueName: \"kubernetes.io/projected/2716609d-ca4a-4bdf-8d24-3f06060a01e4-kube-api-access-ttsrw\") pod \"ai-gateway-controller-656bfc8bbd-llxlr\" (UID: \"2716609d-ca4a-4bdf-8d24-3f06060a01e4\") " pod="opendatahub/ai-gateway-controller-656bfc8bbd-llxlr" Nov 03 20:00:08.679852 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:08.679469 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2716609d-ca4a-4bdf-8d24-3f06060a01e4-metrics-certs\") pod \"ai-gateway-controller-656bfc8bbd-llxlr\" (UID: \"2716609d-ca4a-4bdf-8d24-3f06060a01e4\") " pod="opendatahub/ai-gateway-controller-656bfc8bbd-llxlr" Nov 03 20:00:08.681977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:08.681955 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2716609d-ca4a-4bdf-8d24-3f06060a01e4-metrics-certs\") pod \"ai-gateway-controller-656bfc8bbd-llxlr\" (UID: \"2716609d-ca4a-4bdf-8d24-3f06060a01e4\") " pod="opendatahub/ai-gateway-controller-656bfc8bbd-llxlr" Nov 03 20:00:08.687264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:08.687242 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttsrw\" (UniqueName: \"kubernetes.io/projected/2716609d-ca4a-4bdf-8d24-3f06060a01e4-kube-api-access-ttsrw\") pod \"ai-gateway-controller-656bfc8bbd-llxlr\" (UID: \"2716609d-ca4a-4bdf-8d24-3f06060a01e4\") " pod="opendatahub/ai-gateway-controller-656bfc8bbd-llxlr" Nov 03 20:00:08.778712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:08.778630 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-656bfc8bbd-llxlr" Nov 03 20:00:09.050749 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:09.050720 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-656bfc8bbd-llxlr"] Nov 03 20:00:09.052474 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:00:09.052445 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2716609d_ca4a_4bdf_8d24_3f06060a01e4.slice/crio-0a4be3c1191c1644bd693cf272452d5c0b3df6f81514f4bab40b13767828a6e6 WatchSource:0}: Error finding container 0a4be3c1191c1644bd693cf272452d5c0b3df6f81514f4bab40b13767828a6e6: Status 404 returned error can't find the container with id 0a4be3c1191c1644bd693cf272452d5c0b3df6f81514f4bab40b13767828a6e6 Nov 03 20:00:09.667073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:09.667037 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-656bfc8bbd-llxlr" event={"ID":"2716609d-ca4a-4bdf-8d24-3f06060a01e4","Type":"ContainerStarted","Data":"0a4be3c1191c1644bd693cf272452d5c0b3df6f81514f4bab40b13767828a6e6"} Nov 03 20:00:12.686723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:12.686690 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-656bfc8bbd-llxlr" event={"ID":"2716609d-ca4a-4bdf-8d24-3f06060a01e4","Type":"ContainerStarted","Data":"25d6edf509c47267a5f60a3da49b582af85ee6180bac3924ff9f4158f157034c"} Nov 03 20:00:12.687145 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:12.686761 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-656bfc8bbd-llxlr" Nov 03 20:00:12.703505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:12.703456 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-656bfc8bbd-llxlr" podStartSLOduration=1.1744462119999999 podStartE2EDuration="4.703440466s" totalImagesPullingTime="3.528994254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:12.701726702 +0000 UTC m=+1242.664105143" watchObservedRunningTime="2026-11-03 20:00:12.703440466 +0000 UTC m=+1242.665818904" Nov 03 20:00:18.175786 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:18.175745 5111 scope.go:122] "RemoveContainer" containerID="4a5e974b5a3b4f645cb0cfb8ba9c47d07ed86b871ae08966c8811e0afd768859" Nov 03 20:00:18.718314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:18.718217 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" event={"ID":"6c233789-9149-4c70-871d-8f3b1fe188e3","Type":"ContainerStarted","Data":"af12930213451b199b676637afa149cdb09e4a0b1d552952e2c27e735d09aae7"} Nov 03 20:00:19.725199 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:19.725162 5111 generic.go:356] "Generic (PLEG): container finished" podID="6c233789-9149-4c70-871d-8f3b1fe188e3" containerID="af12930213451b199b676637afa149cdb09e4a0b1d552952e2c27e735d09aae7" exitCode=60 Nov 03 20:00:19.725677 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:19.725221 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" event={"ID":"6c233789-9149-4c70-871d-8f3b1fe188e3","Type":"ContainerDied","Data":"af12930213451b199b676637afa149cdb09e4a0b1d552952e2c27e735d09aae7"} Nov 03 20:00:19.725677 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:19.725279 5111 scope.go:122] "RemoveContainer" containerID="4a5e974b5a3b4f645cb0cfb8ba9c47d07ed86b871ae08966c8811e0afd768859" Nov 03 20:00:19.725677 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:19.725672 5111 scope.go:122] "RemoveContainer" containerID="af12930213451b199b676637afa149cdb09e4a0b1d552952e2c27e735d09aae7" Nov 03 20:00:19.725952 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:00:19.725931 5111 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895600-cxfdp_odh-ai-gateway-infra(6c233789-9149-4c70-871d-8f3b1fe188e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" podUID="6c233789-9149-4c70-871d-8f3b1fe188e3" Nov 03 20:00:19.745338 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:19.745307 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp"] Nov 03 20:00:20.878627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:20.878602 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" Nov 03 20:00:21.005943 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:21.005860 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c233789-9149-4c70-871d-8f3b1fe188e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c233789-9149-4c70-871d-8f3b1fe188e3-maas-api-service-ca\") pod \"6c233789-9149-4c70-871d-8f3b1fe188e3\" (UID: \"6c233789-9149-4c70-871d-8f3b1fe188e3\") " Nov 03 20:00:21.006097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:21.005943 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c233789-9149-4c70-871d-8f3b1fe188e3-kube-api-access-7gcvm\" (UniqueName: \"kubernetes.io/projected/6c233789-9149-4c70-871d-8f3b1fe188e3-kube-api-access-7gcvm\") pod \"6c233789-9149-4c70-871d-8f3b1fe188e3\" (UID: \"6c233789-9149-4c70-871d-8f3b1fe188e3\") " Nov 03 20:00:21.006668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:21.006644 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c233789-9149-4c70-871d-8f3b1fe188e3-maas-api-service-ca" pod "6c233789-9149-4c70-871d-8f3b1fe188e3" (UID: "6c233789-9149-4c70-871d-8f3b1fe188e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:21.008313 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:21.008287 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c233789-9149-4c70-871d-8f3b1fe188e3-kube-api-access-7gcvm" pod "6c233789-9149-4c70-871d-8f3b1fe188e3" (UID: "6c233789-9149-4c70-871d-8f3b1fe188e3"). InnerVolumeSpecName "kube-api-access-7gcvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:21.107710 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:21.107665 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c233789-9149-4c70-871d-8f3b1fe188e3-maas-api-service-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:21.107710 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:21.107704 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gcvm\" (UniqueName: \"kubernetes.io/projected/6c233789-9149-4c70-871d-8f3b1fe188e3-kube-api-access-7gcvm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:21.747207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:21.747171 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" event={"ID":"6c233789-9149-4c70-871d-8f3b1fe188e3","Type":"ContainerDied","Data":"f2ee641c3f73200d0a5f023d3862abb85b8353863fb3e6dcd17724479aa143b8"} Nov 03 20:00:21.747380 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:21.747190 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp" Nov 03 20:00:21.747380 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:21.747256 5111 scope.go:122] "RemoveContainer" containerID="af12930213451b199b676637afa149cdb09e4a0b1d552952e2c27e735d09aae7" Nov 03 20:00:21.770776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:21.770748 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp"] Nov 03 20:00:21.776119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:21.776095 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-cxfdp"] Nov 03 20:00:22.181437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:22.181404 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c233789-9149-4c70-871d-8f3b1fe188e3" path="/var/lib/kubelet/pods/6c233789-9149-4c70-871d-8f3b1fe188e3/volumes" Nov 03 20:00:23.693585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:23.693557 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-656bfc8bbd-llxlr" Nov 03 20:00:35.062440 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:00:35.062401 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:00:51.768919 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.768882 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx"] Nov 03 20:00:51.769474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.769354 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6c233789-9149-4c70-871d-8f3b1fe188e3" containerName="cleanup" Nov 03 20:00:51.769474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.769367 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c233789-9149-4c70-871d-8f3b1fe188e3" containerName="cleanup" Nov 03 20:00:51.769474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.769387 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6c233789-9149-4c70-871d-8f3b1fe188e3" containerName="cleanup" Nov 03 20:00:51.769474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.769396 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c233789-9149-4c70-871d-8f3b1fe188e3" containerName="cleanup" Nov 03 20:00:51.769474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.769406 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6c233789-9149-4c70-871d-8f3b1fe188e3" containerName="cleanup" Nov 03 20:00:51.769474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.769411 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c233789-9149-4c70-871d-8f3b1fe188e3" containerName="cleanup" Nov 03 20:00:51.769765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.769515 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c233789-9149-4c70-871d-8f3b1fe188e3" containerName="cleanup" Nov 03 20:00:51.769765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.769556 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c233789-9149-4c70-871d-8f3b1fe188e3" containerName="cleanup" Nov 03 20:00:51.769765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.769577 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c233789-9149-4c70-871d-8f3b1fe188e3" containerName="cleanup" Nov 03 20:00:51.772599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.772579 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:51.798761 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.798733 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b42aea-openshift-default-dockercfg-7bhzv\"" Nov 03 20:00:51.799774 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.799744 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx"] Nov 03 20:00:51.847523 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.847489 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz"] Nov 03 20:00:51.855505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.855483 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:51.860710 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.860683 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-5348ca-openshift-default-dockercfg-jql62\"" Nov 03 20:00:51.863510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.863486 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz"] Nov 03 20:00:51.907932 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.907905 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:51.908071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.907962 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jm7s\" (UniqueName: \"kubernetes.io/projected/97388cfd-a7f0-425d-946b-bf0181f548b0-kube-api-access-5jm7s\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:51.908118 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.908090 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:51.908173 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.908136 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:51.908288 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.908271 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:51.908347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.908298 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97388cfd-a7f0-425d-946b-bf0181f548b0-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:51.908347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.908328 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:51.908436 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.908412 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:51.908470 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:51.908458 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.009870 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.009834 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.010056 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.009886 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jm7s\" (UniqueName: \"kubernetes.io/projected/97388cfd-a7f0-425d-946b-bf0181f548b0-kube-api-access-5jm7s\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.010056 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.009920 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.010056 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.009947 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.010056 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.009986 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.010056 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.010020 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.010056 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.010039 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.010056 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.010054 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.010435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.010078 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppbr4\" (UniqueName: \"kubernetes.io/projected/1e1df657-2ab6-4ae7-80e5-0c6358becefd-kube-api-access-ppbr4\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.010435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.010113 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.010435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.010130 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97388cfd-a7f0-425d-946b-bf0181f548b0-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.010435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.010146 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.010435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.010170 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.010435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.010192 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.010435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.010210 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.010435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.010244 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.010435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.010259 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.010435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.010256 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.010435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.010277 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.011697 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.011668 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97388cfd-a7f0-425d-946b-bf0181f548b0-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.012201 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.012178 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.012560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.012405 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.012560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.012550 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.013155 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.013138 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.015812 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.014892 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.015937 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.015760 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt"] Nov 03 20:00:52.019935 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.019867 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.022181 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.022154 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jm7s\" (UniqueName: \"kubernetes.io/projected/97388cfd-a7f0-425d-946b-bf0181f548b0-kube-api-access-5jm7s\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.022593 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.022571 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.034582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.034561 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt"] Nov 03 20:00:52.055076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.055051 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-120fe3-openshift-default-dockercfg-7jshr\"" Nov 03 20:00:52.099496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.099466 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:52.112149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112123 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.112271 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112162 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.112271 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112218 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.112271 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112247 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.112431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112288 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.112431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112313 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.112431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112343 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm2f7\" (UniqueName: \"kubernetes.io/projected/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-kube-api-access-wm2f7\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.112431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112392 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.112431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112419 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.112716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112443 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.112716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112477 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppbr4\" (UniqueName: \"kubernetes.io/projected/1e1df657-2ab6-4ae7-80e5-0c6358becefd-kube-api-access-ppbr4\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.112716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112516 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.112716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112589 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.112716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112641 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.112716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112671 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.112995 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112728 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.112995 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112793 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.112995 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.112817 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.114171 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.113503 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.114171 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.113784 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.114171 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.113988 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.114171 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.114105 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.114450 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.114405 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.117026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.117002 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.120935 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.120910 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.122805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.122778 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppbr4\" (UniqueName: \"kubernetes.io/projected/1e1df657-2ab6-4ae7-80e5-0c6358becefd-kube-api-access-ppbr4\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.124327 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.124303 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.173053 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.172999 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:52.213823 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.213789 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.213969 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.213859 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.213969 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.213889 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.213969 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.213927 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.213969 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.213957 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm2f7\" (UniqueName: \"kubernetes.io/projected/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-kube-api-access-wm2f7\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.214175 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.214071 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.214175 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.214109 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.214175 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.214134 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.214332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.214177 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.214922 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.214888 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.215157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.215136 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.215157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.215148 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.215421 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.215399 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.217460 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.215755 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.217574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.217457 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.220235 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.219944 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.224574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.224400 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.224928 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.224905 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm2f7\" (UniqueName: \"kubernetes.io/projected/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-kube-api-access-wm2f7\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.352415 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.352339 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:52.421555 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:00:52.420352 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97388cfd_a7f0_425d_946b_bf0181f548b0.slice/crio-c594298b69cb59dc05a52927e3f736e4e5c1d3c4a8f97241049049b443761b80 WatchSource:0}: Error finding container c594298b69cb59dc05a52927e3f736e4e5c1d3c4a8f97241049049b443761b80: Status 404 returned error can't find the container with id c594298b69cb59dc05a52927e3f736e4e5c1d3c4a8f97241049049b443761b80 Nov 03 20:00:52.430553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.427895 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:00:52.430553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.427995 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:00:52.430553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.428035 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:00:52.438846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.433358 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx"] Nov 03 20:00:52.494960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.494118 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz"] Nov 03 20:00:52.498220 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:00:52.498190 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e1df657_2ab6_4ae7_80e5_0c6358becefd.slice/crio-6ac563082718fbda412a6933d80060ffc5731b52c93966780e29cc3c0e139c22 WatchSource:0}: Error finding container 6ac563082718fbda412a6933d80060ffc5731b52c93966780e29cc3c0e139c22: Status 404 returned error can't find the container with id 6ac563082718fbda412a6933d80060ffc5731b52c93966780e29cc3c0e139c22 Nov 03 20:00:52.500831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.500804 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:00:52.500908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.500865 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:00:52.500908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.500893 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:00:52.703615 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.703513 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt"] Nov 03 20:00:52.709194 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:00:52.709166 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2114d1_4f1c_42a1_b0db_02c8f3c5386c.slice/crio-9864e8ed8a9441811ee67ee765d0cf05fdef721bbf15c0be08fc24eacdf54b61 WatchSource:0}: Error finding container 9864e8ed8a9441811ee67ee765d0cf05fdef721bbf15c0be08fc24eacdf54b61: Status 404 returned error can't find the container with id 9864e8ed8a9441811ee67ee765d0cf05fdef721bbf15c0be08fc24eacdf54b61 Nov 03 20:00:52.711720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.711687 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:00:52.711837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.711758 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:00:52.711837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.711789 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:00:52.919442 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.919405 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" event={"ID":"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c","Type":"ContainerStarted","Data":"cb89a204892d6df55eedef7c1985ec05d283c25586c3c8caf7487af583d70821"} Nov 03 20:00:52.919889 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.919468 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" event={"ID":"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c","Type":"ContainerStarted","Data":"9864e8ed8a9441811ee67ee765d0cf05fdef721bbf15c0be08fc24eacdf54b61"} Nov 03 20:00:52.921605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.921569 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" event={"ID":"97388cfd-a7f0-425d-946b-bf0181f548b0","Type":"ContainerStarted","Data":"7066ba48898c82f9859b2cf5b411008139e2782c58d7ee14dc575085b7bcee1c"} Nov 03 20:00:52.921740 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.921628 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" event={"ID":"97388cfd-a7f0-425d-946b-bf0181f548b0","Type":"ContainerStarted","Data":"c594298b69cb59dc05a52927e3f736e4e5c1d3c4a8f97241049049b443761b80"} Nov 03 20:00:52.923230 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.923206 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" event={"ID":"1e1df657-2ab6-4ae7-80e5-0c6358becefd","Type":"ContainerStarted","Data":"25e2de8c87631905c805661111eadd55007460a31d6054d6fa01c651f13302f7"} Nov 03 20:00:52.923315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.923246 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" event={"ID":"1e1df657-2ab6-4ae7-80e5-0c6358becefd","Type":"ContainerStarted","Data":"6ac563082718fbda412a6933d80060ffc5731b52c93966780e29cc3c0e139c22"} Nov 03 20:00:52.939545 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.939485 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" podStartSLOduration=1.939470413 podStartE2EDuration="1.939470413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:52.937290757 +0000 UTC m=+1282.899669208" watchObservedRunningTime="2026-11-03 20:00:52.939470413 +0000 UTC m=+1282.901848852" Nov 03 20:00:52.955735 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.955641 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" podStartSLOduration=1.955626014 podStartE2EDuration="1.955626014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:52.95483622 +0000 UTC m=+1282.917214658" watchObservedRunningTime="2026-11-03 20:00:52.955626014 +0000 UTC m=+1282.918004452" Nov 03 20:00:52.975446 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:52.975395 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" podStartSLOduration=1.9753795379999999 podStartE2EDuration="1.975379538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:52.972726088 +0000 UTC m=+1282.935104531" watchObservedRunningTime="2026-11-03 20:00:52.975379538 +0000 UTC m=+1282.937757976" Nov 03 20:00:53.100339 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:53.100254 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:53.116066 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:53.116030 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:53.173848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:53.173818 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:53.178918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:53.178894 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:53.352937 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:53.352865 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:53.358155 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:53.358133 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:53.929259 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:53.928748 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:53.929259 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:53.928811 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:53.929259 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:53.928834 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:53.929814 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:53.929719 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:00:53.929999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:53.929975 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:00:53.930242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:53.930228 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:00:57.034659 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.034618 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2"] Nov 03 20:00:57.038612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.038588 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" Nov 03 20:00:57.045566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.045543 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4"] Nov 03 20:00:57.047482 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.047463 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-120fe3-dockercfg-xgcj2\"" Nov 03 20:00:57.048423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.048402 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-120fe3\"" Nov 03 20:00:57.049728 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.049709 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" Nov 03 20:00:57.050615 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.050593 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2"] Nov 03 20:00:57.062873 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.062804 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4"] Nov 03 20:00:57.174327 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.174294 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqc85\" (UniqueName: \"kubernetes.io/projected/9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4-kube-api-access-zqc85\") pod \"payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4\" (UID: \"9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" Nov 03 20:00:57.174686 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.174660 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mx2q\" (UniqueName: \"kubernetes.io/projected/267cf330-341c-40a3-9b8e-695d9273786a-kube-api-access-8mx2q\") pod \"payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2\" (UID: \"267cf330-341c-40a3-9b8e-695d9273786a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" Nov 03 20:00:57.174812 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.174714 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267cf330-341c-40a3-9b8e-695d9273786a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2\" (UID: \"267cf330-341c-40a3-9b8e-695d9273786a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" Nov 03 20:00:57.174874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.174825 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4\" (UID: \"9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" Nov 03 20:00:57.276309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.276270 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4\" (UID: \"9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" Nov 03 20:00:57.276690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.276638 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqc85\" (UniqueName: \"kubernetes.io/projected/9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4-kube-api-access-zqc85\") pod \"payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4\" (UID: \"9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" Nov 03 20:00:57.276899 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.276880 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mx2q\" (UniqueName: \"kubernetes.io/projected/267cf330-341c-40a3-9b8e-695d9273786a-kube-api-access-8mx2q\") pod \"payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2\" (UID: \"267cf330-341c-40a3-9b8e-695d9273786a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" Nov 03 20:00:57.277038 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.277022 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267cf330-341c-40a3-9b8e-695d9273786a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2\" (UID: \"267cf330-341c-40a3-9b8e-695d9273786a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" Nov 03 20:00:57.278097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.278072 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4\" (UID: \"9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" Nov 03 20:00:57.279883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.279860 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267cf330-341c-40a3-9b8e-695d9273786a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2\" (UID: \"267cf330-341c-40a3-9b8e-695d9273786a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" Nov 03 20:00:57.287248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.287199 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqc85\" (UniqueName: \"kubernetes.io/projected/9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4-kube-api-access-zqc85\") pod \"payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4\" (UID: \"9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" Nov 03 20:00:57.287428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.287405 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mx2q\" (UniqueName: \"kubernetes.io/projected/267cf330-341c-40a3-9b8e-695d9273786a-kube-api-access-8mx2q\") pod \"payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2\" (UID: \"267cf330-341c-40a3-9b8e-695d9273786a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" Nov 03 20:00:57.354431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.354397 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" Nov 03 20:00:57.368059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.368026 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" Nov 03 20:00:57.637253 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.637223 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2"] Nov 03 20:00:57.638198 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:00:57.638169 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267cf330_341c_40a3_9b8e_695d9273786a.slice/crio-af74551275fce5acf6b175627cb657f1c0062fd4e89dd06478ef4b6afddce388 WatchSource:0}: Error finding container af74551275fce5acf6b175627cb657f1c0062fd4e89dd06478ef4b6afddce388: Status 404 returned error can't find the container with id af74551275fce5acf6b175627cb657f1c0062fd4e89dd06478ef4b6afddce388 Nov 03 20:00:57.667126 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.666825 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4"] Nov 03 20:00:57.674255 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:00:57.674213 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9619ccf6_8bbb_4c1d_bf4d_675d674ad3d4.slice/crio-fb712a19ab08243c50e4f2ceb9a1c660bf776371a39cb1e1f4161be03fe22393 WatchSource:0}: Error finding container fb712a19ab08243c50e4f2ceb9a1c660bf776371a39cb1e1f4161be03fe22393: Status 404 returned error can't find the container with id fb712a19ab08243c50e4f2ceb9a1c660bf776371a39cb1e1f4161be03fe22393 Nov 03 20:00:57.874171 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.874137 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx"] Nov 03 20:00:57.879397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.879370 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" Nov 03 20:00:57.887098 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.887039 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx"] Nov 03 20:00:57.950585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.950526 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" event={"ID":"9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4","Type":"ContainerStarted","Data":"fb712a19ab08243c50e4f2ceb9a1c660bf776371a39cb1e1f4161be03fe22393"} Nov 03 20:00:57.951825 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.951798 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" event={"ID":"267cf330-341c-40a3-9b8e-695d9273786a","Type":"ContainerStarted","Data":"af74551275fce5acf6b175627cb657f1c0062fd4e89dd06478ef4b6afddce388"} Nov 03 20:00:57.998746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.997632 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a39316d-9016-42b8-93d7-73296fd2bfde-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx\" (UID: \"6a39316d-9016-42b8-93d7-73296fd2bfde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" Nov 03 20:00:57.998746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.997769 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zqgz\" (UniqueName: \"kubernetes.io/projected/6a39316d-9016-42b8-93d7-73296fd2bfde-kube-api-access-7zqgz\") pod \"payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx\" (UID: \"6a39316d-9016-42b8-93d7-73296fd2bfde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" Nov 03 20:00:57.999428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.999286 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-5348ca-dockercfg-x698f\"" Nov 03 20:00:57.999428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:57.999401 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-5348ca\"" Nov 03 20:00:58.099243 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.099209 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a39316d-9016-42b8-93d7-73296fd2bfde-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx\" (UID: \"6a39316d-9016-42b8-93d7-73296fd2bfde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" Nov 03 20:00:58.099769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.099323 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zqgz\" (UniqueName: \"kubernetes.io/projected/6a39316d-9016-42b8-93d7-73296fd2bfde-kube-api-access-7zqgz\") pod \"payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx\" (UID: \"6a39316d-9016-42b8-93d7-73296fd2bfde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" Nov 03 20:00:58.100178 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.100156 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a39316d-9016-42b8-93d7-73296fd2bfde-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx\" (UID: \"6a39316d-9016-42b8-93d7-73296fd2bfde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" Nov 03 20:00:58.107724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.107699 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zqgz\" (UniqueName: \"kubernetes.io/projected/6a39316d-9016-42b8-93d7-73296fd2bfde-kube-api-access-7zqgz\") pod \"payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx\" (UID: \"6a39316d-9016-42b8-93d7-73296fd2bfde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" Nov 03 20:00:58.197444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.197368 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" Nov 03 20:00:58.309933 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.309895 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j"] Nov 03 20:00:58.317315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.317285 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" Nov 03 20:00:58.327234 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.327203 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-b42aea\"" Nov 03 20:00:58.343259 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.343220 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j"] Nov 03 20:00:58.404004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.403735 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n8pn\" (UniqueName: \"kubernetes.io/projected/8d4c64f2-4b08-4847-ade9-8035b3f7853c-kube-api-access-9n8pn\") pod \"payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j\" (UID: \"8d4c64f2-4b08-4847-ade9-8035b3f7853c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" Nov 03 20:00:58.404004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.403858 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4c64f2-4b08-4847-ade9-8035b3f7853c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j\" (UID: \"8d4c64f2-4b08-4847-ade9-8035b3f7853c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" Nov 03 20:00:58.494359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.494327 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx"] Nov 03 20:00:58.504707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.504677 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n8pn\" (UniqueName: \"kubernetes.io/projected/8d4c64f2-4b08-4847-ade9-8035b3f7853c-kube-api-access-9n8pn\") pod \"payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j\" (UID: \"8d4c64f2-4b08-4847-ade9-8035b3f7853c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" Nov 03 20:00:58.504815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.504785 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4c64f2-4b08-4847-ade9-8035b3f7853c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j\" (UID: \"8d4c64f2-4b08-4847-ade9-8035b3f7853c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" Nov 03 20:00:58.505664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.505636 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4c64f2-4b08-4847-ade9-8035b3f7853c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j\" (UID: \"8d4c64f2-4b08-4847-ade9-8035b3f7853c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" Nov 03 20:00:58.513473 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.513452 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n8pn\" (UniqueName: \"kubernetes.io/projected/8d4c64f2-4b08-4847-ade9-8035b3f7853c-kube-api-access-9n8pn\") pod \"payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j\" (UID: \"8d4c64f2-4b08-4847-ade9-8035b3f7853c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" Nov 03 20:00:58.637405 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.637377 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" Nov 03 20:00:58.933091 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.933021 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j"] Nov 03 20:00:58.948246 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:00:58.948210 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4c64f2_4b08_4847_ade9_8035b3f7853c.slice/crio-330ffae8d76b91261c586f5500b249f6ebcc7831923b010d856935bf0fb4e5c9 WatchSource:0}: Error finding container 330ffae8d76b91261c586f5500b249f6ebcc7831923b010d856935bf0fb4e5c9: Status 404 returned error can't find the container with id 330ffae8d76b91261c586f5500b249f6ebcc7831923b010d856935bf0fb4e5c9 Nov 03 20:00:58.959679 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.959641 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" event={"ID":"8d4c64f2-4b08-4847-ade9-8035b3f7853c","Type":"ContainerStarted","Data":"330ffae8d76b91261c586f5500b249f6ebcc7831923b010d856935bf0fb4e5c9"} Nov 03 20:00:58.961073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:58.961049 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" event={"ID":"6a39316d-9016-42b8-93d7-73296fd2bfde","Type":"ContainerStarted","Data":"a90f81b7bbf1c018220cdb05dc5523ea1d093f829fe032e60b9ae783ec177090"} Nov 03 20:00:59.366278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.366195 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc"] Nov 03 20:00:59.374062 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.374039 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" Nov 03 20:00:59.380265 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.379770 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-b87733\"" Nov 03 20:00:59.389312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.386420 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc"] Nov 03 20:00:59.406244 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.406213 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc"] Nov 03 20:00:59.410791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.410764 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" Nov 03 20:00:59.418355 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.418331 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-b87733-dockercfg-xdztp\"" Nov 03 20:00:59.418888 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.418858 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97dad6bc-5f5e-49cf-9fde-1470e77512f0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc\" (UID: \"97dad6bc-5f5e-49cf-9fde-1470e77512f0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" Nov 03 20:00:59.419059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.418956 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcv22\" (UniqueName: \"kubernetes.io/projected/105f9648-6b47-477c-afb1-a41d07095e1b-kube-api-access-pcv22\") pod \"payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc\" (UID: \"105f9648-6b47-477c-afb1-a41d07095e1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" Nov 03 20:00:59.419161 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.419081 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn88p\" (UniqueName: \"kubernetes.io/projected/97dad6bc-5f5e-49cf-9fde-1470e77512f0-kube-api-access-xn88p\") pod \"payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc\" (UID: \"97dad6bc-5f5e-49cf-9fde-1470e77512f0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" Nov 03 20:00:59.419252 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.419164 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105f9648-6b47-477c-afb1-a41d07095e1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc\" (UID: \"105f9648-6b47-477c-afb1-a41d07095e1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" Nov 03 20:00:59.423589 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.423567 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc"] Nov 03 20:00:59.521697 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.521066 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcv22\" (UniqueName: \"kubernetes.io/projected/105f9648-6b47-477c-afb1-a41d07095e1b-kube-api-access-pcv22\") pod \"payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc\" (UID: \"105f9648-6b47-477c-afb1-a41d07095e1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" Nov 03 20:00:59.521697 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.521173 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn88p\" (UniqueName: \"kubernetes.io/projected/97dad6bc-5f5e-49cf-9fde-1470e77512f0-kube-api-access-xn88p\") pod \"payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc\" (UID: \"97dad6bc-5f5e-49cf-9fde-1470e77512f0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" Nov 03 20:00:59.521697 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.521247 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105f9648-6b47-477c-afb1-a41d07095e1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc\" (UID: \"105f9648-6b47-477c-afb1-a41d07095e1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" Nov 03 20:00:59.521697 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.521281 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97dad6bc-5f5e-49cf-9fde-1470e77512f0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc\" (UID: \"97dad6bc-5f5e-49cf-9fde-1470e77512f0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" Nov 03 20:00:59.560599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.548172 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105f9648-6b47-477c-afb1-a41d07095e1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc\" (UID: \"105f9648-6b47-477c-afb1-a41d07095e1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" Nov 03 20:00:59.562355 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.562292 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97dad6bc-5f5e-49cf-9fde-1470e77512f0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc\" (UID: \"97dad6bc-5f5e-49cf-9fde-1470e77512f0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" Nov 03 20:00:59.574190 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.574160 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn88p\" (UniqueName: \"kubernetes.io/projected/97dad6bc-5f5e-49cf-9fde-1470e77512f0-kube-api-access-xn88p\") pod \"payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc\" (UID: \"97dad6bc-5f5e-49cf-9fde-1470e77512f0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" Nov 03 20:00:59.584496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.584446 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcv22\" (UniqueName: \"kubernetes.io/projected/105f9648-6b47-477c-afb1-a41d07095e1b-kube-api-access-pcv22\") pod \"payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc\" (UID: \"105f9648-6b47-477c-afb1-a41d07095e1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" Nov 03 20:00:59.700140 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.699814 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" Nov 03 20:00:59.736804 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:00:59.736506 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" Nov 03 20:01:00.101312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.101274 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5"] Nov 03 20:01:00.125341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.123956 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc"] Nov 03 20:01:00.125341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.124024 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc"] Nov 03 20:01:00.125341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.124190 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" Nov 03 20:01:00.125341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.125245 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5"] Nov 03 20:01:00.150099 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.150042 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55381c03-06d5-4cdb-841f-265b16b1d8f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5\" (UID: \"55381c03-06d5-4cdb-841f-265b16b1d8f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" Nov 03 20:01:00.150259 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.150153 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw9hd\" (UniqueName: \"kubernetes.io/projected/55381c03-06d5-4cdb-841f-265b16b1d8f8-kube-api-access-sw9hd\") pod \"payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5\" (UID: \"55381c03-06d5-4cdb-841f-265b16b1d8f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" Nov 03 20:01:00.159592 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.159527 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-0ed729-dockercfg-fndg4\"" Nov 03 20:01:00.219515 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.219482 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-0ed729\"" Nov 03 20:01:00.252816 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.252198 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw9hd\" (UniqueName: \"kubernetes.io/projected/55381c03-06d5-4cdb-841f-265b16b1d8f8-kube-api-access-sw9hd\") pod \"payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5\" (UID: \"55381c03-06d5-4cdb-841f-265b16b1d8f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" Nov 03 20:01:00.252816 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.252405 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55381c03-06d5-4cdb-841f-265b16b1d8f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5\" (UID: \"55381c03-06d5-4cdb-841f-265b16b1d8f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" Nov 03 20:01:00.254051 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.254028 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55381c03-06d5-4cdb-841f-265b16b1d8f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5\" (UID: \"55381c03-06d5-4cdb-841f-265b16b1d8f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" Nov 03 20:01:00.270323 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.269814 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw9hd\" (UniqueName: \"kubernetes.io/projected/55381c03-06d5-4cdb-841f-265b16b1d8f8-kube-api-access-sw9hd\") pod \"payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5\" (UID: \"55381c03-06d5-4cdb-841f-265b16b1d8f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" Nov 03 20:01:00.468660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.468425 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" Nov 03 20:01:00.528204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.528167 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6"] Nov 03 20:01:00.547216 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.546145 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" Nov 03 20:01:00.551603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.551512 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6"] Nov 03 20:01:00.583633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.583598 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbg5j\" (UniqueName: \"kubernetes.io/projected/67f55725-b707-46f2-9cf5-8932b47df3b8-kube-api-access-nbg5j\") pod \"payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6\" (UID: \"67f55725-b707-46f2-9cf5-8932b47df3b8\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" Nov 03 20:01:00.583738 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.583685 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f55725-b707-46f2-9cf5-8932b47df3b8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6\" (UID: \"67f55725-b707-46f2-9cf5-8932b47df3b8\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" Nov 03 20:01:00.616572 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.608791 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-ab1e46-dockercfg-cdkjl\"" Nov 03 20:01:00.616572 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.608964 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-ab1e46\"" Nov 03 20:01:00.684676 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.684621 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbg5j\" (UniqueName: \"kubernetes.io/projected/67f55725-b707-46f2-9cf5-8932b47df3b8-kube-api-access-nbg5j\") pod \"payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6\" (UID: \"67f55725-b707-46f2-9cf5-8932b47df3b8\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" Nov 03 20:01:00.685554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.685520 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f55725-b707-46f2-9cf5-8932b47df3b8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6\" (UID: \"67f55725-b707-46f2-9cf5-8932b47df3b8\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" Nov 03 20:01:00.686251 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.686230 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f55725-b707-46f2-9cf5-8932b47df3b8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6\" (UID: \"67f55725-b707-46f2-9cf5-8932b47df3b8\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" Nov 03 20:01:00.706806 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.706780 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbg5j\" (UniqueName: \"kubernetes.io/projected/67f55725-b707-46f2-9cf5-8932b47df3b8-kube-api-access-nbg5j\") pod \"payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6\" (UID: \"67f55725-b707-46f2-9cf5-8932b47df3b8\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" Nov 03 20:01:00.780022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.779885 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5"] Nov 03 20:01:00.790495 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:01:00.790458 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55381c03_06d5_4cdb_841f_265b16b1d8f8.slice/crio-0dfbbdac1b08627a09ec804b18f557c0bfe6e0469d8b1bda02cc945ae13ce582 WatchSource:0}: Error finding container 0dfbbdac1b08627a09ec804b18f557c0bfe6e0469d8b1bda02cc945ae13ce582: Status 404 returned error can't find the container with id 0dfbbdac1b08627a09ec804b18f557c0bfe6e0469d8b1bda02cc945ae13ce582 Nov 03 20:01:00.875014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.874963 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" Nov 03 20:01:00.999054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:00.999015 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" event={"ID":"105f9648-6b47-477c-afb1-a41d07095e1b","Type":"ContainerStarted","Data":"a3ce06a49d757ea93f704c2792784b259f23ceee6e7dabead092e689ca10efbd"} Nov 03 20:01:01.005884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:01.005852 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" event={"ID":"97dad6bc-5f5e-49cf-9fde-1470e77512f0","Type":"ContainerStarted","Data":"d1932f73a82dea91cacab9069e70dd7d4b9bb1463c0455773417365c9f107b40"} Nov 03 20:01:01.012741 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:01.012674 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" event={"ID":"55381c03-06d5-4cdb-841f-265b16b1d8f8","Type":"ContainerStarted","Data":"0dfbbdac1b08627a09ec804b18f557c0bfe6e0469d8b1bda02cc945ae13ce582"} Nov 03 20:01:01.283028 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:01.282989 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6"] Nov 03 20:01:02.024906 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:02.024875 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" event={"ID":"67f55725-b707-46f2-9cf5-8932b47df3b8","Type":"ContainerStarted","Data":"4d8d5230cb41f2082eab23bda1f636c298356a2228c6b14b08fd2fab39d2bd1c"} Nov 03 20:01:03.041044 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:03.040909 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" event={"ID":"8d4c64f2-4b08-4847-ade9-8035b3f7853c","Type":"ContainerStarted","Data":"a73901a8b96750605632061a49b742f4d6e607b3571d12dd648a954dbb923474"} Nov 03 20:01:03.048364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:03.047623 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" event={"ID":"67f55725-b707-46f2-9cf5-8932b47df3b8","Type":"ContainerStarted","Data":"8306cc4bbe5d8c8e75626e264e1ab35c1dac9aa40f6cface6df09adedd14978a"} Nov 03 20:01:03.048364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:03.048237 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" Nov 03 20:01:03.056170 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:03.056117 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" event={"ID":"267cf330-341c-40a3-9b8e-695d9273786a","Type":"ContainerStarted","Data":"c6bb762c312cf8de715601a690653b875f152059577fa67f341b4ad7d182a7be"} Nov 03 20:01:03.056354 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:03.056321 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" Nov 03 20:01:03.060375 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:03.060329 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" podStartSLOduration=1.792863281 podStartE2EDuration="5.060313493s" totalImagesPullingTime="3.267450212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:03.059677285 +0000 UTC m=+1293.022055727" watchObservedRunningTime="2026-11-03 20:01:03.060313493 +0000 UTC m=+1293.022691932" Nov 03 20:01:03.066279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:03.066198 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" event={"ID":"105f9648-6b47-477c-afb1-a41d07095e1b","Type":"ContainerStarted","Data":"9642ceac49c2e7bf0ff1e5dd96112fa9ce84a4c6e7dc79dbe75cff787f3f7dd7"} Nov 03 20:01:03.067170 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:03.067113 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" Nov 03 20:01:03.070249 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:03.069722 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" event={"ID":"97dad6bc-5f5e-49cf-9fde-1470e77512f0","Type":"ContainerStarted","Data":"fc02b840bf54c11f0cc5cfaf4ca0dda12e7c5bf529c908a7311ad637e8f84ac8"} Nov 03 20:01:03.070249 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:03.070202 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" Nov 03 20:01:03.076936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:03.076897 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" podStartSLOduration=1.6969845289999999 podStartE2EDuration="6.076882257s" totalImagesPullingTime="4.379897728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:03.074888567 +0000 UTC m=+1293.037267007" watchObservedRunningTime="2026-11-03 20:01:03.076882257 +0000 UTC m=+1293.039260697" Nov 03 20:01:03.119100 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:03.119048 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" podStartSLOduration=2.278153803 podStartE2EDuration="3.119035417s" totalImagesPullingTime="840.881614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:03.095803418 +0000 UTC m=+1293.058181857" watchObservedRunningTime="2026-11-03 20:01:03.119035417 +0000 UTC m=+1293.081413855" Nov 03 20:01:03.137932 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:03.137884 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" podStartSLOduration=1.5412240069999998 podStartE2EDuration="4.137870884s" totalImagesPullingTime="2.596646877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:03.117309834 +0000 UTC m=+1293.079688275" watchObservedRunningTime="2026-11-03 20:01:03.137870884 +0000 UTC m=+1293.100249320" Nov 03 20:01:03.139194 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:03.139140 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" podStartSLOduration=1.5557192130000002 podStartE2EDuration="4.139131904s" totalImagesPullingTime="2.583412691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:03.135398224 +0000 UTC m=+1293.097776658" watchObservedRunningTime="2026-11-03 20:01:03.139131904 +0000 UTC m=+1293.101510340" Nov 03 20:01:04.076594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:04.076523 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" event={"ID":"9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4","Type":"ContainerStarted","Data":"1853bc768b19ef71caf8e5968373d6c4b909c1d2edd6a1acdd4ce2b101eac0b6"} Nov 03 20:01:04.077030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:04.076778 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" Nov 03 20:01:04.078351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:04.078318 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" event={"ID":"6a39316d-9016-42b8-93d7-73296fd2bfde","Type":"ContainerStarted","Data":"9bc15d88baa04f72f3f5c7618002198a49029bc1a586788f87a96e1c7d5ed245"} Nov 03 20:01:04.078510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:04.078406 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" Nov 03 20:01:04.080025 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:04.079989 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" event={"ID":"55381c03-06d5-4cdb-841f-265b16b1d8f8","Type":"ContainerStarted","Data":"ad865cc5d91a551c27a0b208cb906f2c38cb9852ca042ddf2739fc65a9c78e42"} Nov 03 20:01:04.080565 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:04.080547 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" Nov 03 20:01:04.080690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:04.080673 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" Nov 03 20:01:04.095831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:04.095789 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" podStartSLOduration=1.370712258 podStartE2EDuration="7.095778169s" totalImagesPullingTime="5.725065911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:04.093902275 +0000 UTC m=+1294.056280714" watchObservedRunningTime="2026-11-03 20:01:04.095778169 +0000 UTC m=+1294.058156607" Nov 03 20:01:04.111233 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:04.111193 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" podStartSLOduration=1.6180462690000001 podStartE2EDuration="4.111182549s" totalImagesPullingTime="2.49313628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:04.109730731 +0000 UTC m=+1294.072109169" watchObservedRunningTime="2026-11-03 20:01:04.111182549 +0000 UTC m=+1294.073560987" Nov 03 20:01:04.129890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:04.129851 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" podStartSLOduration=2.281909294 podStartE2EDuration="7.129840759s" totalImagesPullingTime="4.847931465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:04.127916116 +0000 UTC m=+1294.090294555" watchObservedRunningTime="2026-11-03 20:01:04.129840759 +0000 UTC m=+1294.092219196" Nov 03 20:01:05.054791 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:01:05.054759 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:01:13.307498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.307455 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4"] Nov 03 20:01:13.316643 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.311866 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.323496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.323466 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ab1e46\"/\"default-dockercfg-5t6p5\"" Nov 03 20:01:13.325195 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.325169 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4"] Nov 03 20:01:13.325339 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.325311 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ab1e46\"/\"kube-root-ca.crt\"" Nov 03 20:01:13.325398 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.325180 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ab1e46\"/\"openshift-service-ca.crt\"" Nov 03 20:01:13.373910 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.373831 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ab1e46\"/\"simulator-w5-ab1e46-kserve-self-signed-certs\"" Nov 03 20:01:13.379736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.379692 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-tmp-dir\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.379911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.379745 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-home\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.379911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.379872 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqxpg\" (UniqueName: \"kubernetes.io/projected/afbc0e79-dd6e-4c42-9330-ff7988817e80-kube-api-access-sqxpg\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.380006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.379925 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-model-cache\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.380006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.379981 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-dshm\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.380165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.380138 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afbc0e79-dd6e-4c42-9330-ff7988817e80-tls-certs\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.484207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.482134 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqxpg\" (UniqueName: \"kubernetes.io/projected/afbc0e79-dd6e-4c42-9330-ff7988817e80-kube-api-access-sqxpg\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.484207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.482225 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-model-cache\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.484207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.482291 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-dshm\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.484207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.482369 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afbc0e79-dd6e-4c42-9330-ff7988817e80-tls-certs\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.484207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.482441 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-tmp-dir\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.484207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.482468 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-home\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.484207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.482961 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-home\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.484207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.483556 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-model-cache\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.485639 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.485593 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-tmp-dir\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.495947 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.495889 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afbc0e79-dd6e-4c42-9330-ff7988817e80-tls-certs\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.496660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.496609 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-dshm\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.516555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.512771 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqxpg\" (UniqueName: \"kubernetes.io/projected/afbc0e79-dd6e-4c42-9330-ff7988817e80-kube-api-access-sqxpg\") pod \"simulator-w5-ab1e46-kserve-7988b48557-lt2w4\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:13.628371 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:13.628322 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:14.024295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:14.024213 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4"] Nov 03 20:01:14.029058 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:01:14.029025 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafbc0e79_dd6e_4c42_9330_ff7988817e80.slice/crio-166fdb35a5b3c1f225cae705fdc21d98fbf822d9f570fc77bc76944d093c0e50 WatchSource:0}: Error finding container 166fdb35a5b3c1f225cae705fdc21d98fbf822d9f570fc77bc76944d093c0e50: Status 404 returned error can't find the container with id 166fdb35a5b3c1f225cae705fdc21d98fbf822d9f570fc77bc76944d093c0e50 Nov 03 20:01:14.086571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:14.085201 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" Nov 03 20:01:14.194027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:14.193992 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" event={"ID":"afbc0e79-dd6e-4c42-9330-ff7988817e80","Type":"ContainerStarted","Data":"87a653f7cc8f20e0e426f7ebc307d0f19138d0887301168396d1850f951f60fe"} Nov 03 20:01:14.194027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:14.194034 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" event={"ID":"afbc0e79-dd6e-4c42-9330-ff7988817e80","Type":"ContainerStarted","Data":"166fdb35a5b3c1f225cae705fdc21d98fbf822d9f570fc77bc76944d093c0e50"} Nov 03 20:01:14.629581 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:14.629514 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:15.088418 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:15.088346 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" Nov 03 20:01:15.089453 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:15.089426 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" Nov 03 20:01:15.091874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:15.091853 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" Nov 03 20:01:15.092947 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:15.092339 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" Nov 03 20:01:15.092947 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:15.092627 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" Nov 03 20:01:15.092947 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:15.092672 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" Nov 03 20:01:15.106541 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:15.106493 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podStartSLOduration=2.106480891 podStartE2EDuration="2.106480891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:14.215275314 +0000 UTC m=+1304.177653750" watchObservedRunningTime="2026-11-03 20:01:15.106480891 +0000 UTC m=+1305.068859363" Nov 03 20:01:16.096187 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:16.096149 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" Nov 03 20:01:17.238052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:17.237974 5111 generic.go:356] "Generic (PLEG): container finished" podID="1e1df657-2ab6-4ae7-80e5-0c6358becefd" containerID="25e2de8c87631905c805661111eadd55007460a31d6054d6fa01c651f13302f7" exitCode=0 Nov 03 20:01:17.238602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:17.238050 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" event={"ID":"1e1df657-2ab6-4ae7-80e5-0c6358becefd","Type":"ContainerDied","Data":"25e2de8c87631905c805661111eadd55007460a31d6054d6fa01c651f13302f7"} Nov 03 20:01:17.238602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:17.238594 5111 scope.go:122] "RemoveContainer" containerID="25e2de8c87631905c805661111eadd55007460a31d6054d6fa01c651f13302f7" Nov 03 20:01:17.240038 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:17.240009 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:01:17.240110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:17.240079 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:01:17.240147 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:17.240112 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:01:18.272358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:18.272300 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" event={"ID":"1e1df657-2ab6-4ae7-80e5-0c6358becefd","Type":"ContainerStarted","Data":"98514336e723f7a5c0eac41318036eaa82e8c4bf2ba5ee7fb550ae57a93f240a"} Nov 03 20:01:18.296558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:18.295138 5111 generic.go:356] "Generic (PLEG): container finished" podID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerID="cb89a204892d6df55eedef7c1985ec05d283c25586c3c8caf7487af583d70821" exitCode=0 Nov 03 20:01:18.296558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:18.295221 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" event={"ID":"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c","Type":"ContainerDied","Data":"cb89a204892d6df55eedef7c1985ec05d283c25586c3c8caf7487af583d70821"} Nov 03 20:01:18.296558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:18.295666 5111 scope.go:122] "RemoveContainer" containerID="cb89a204892d6df55eedef7c1985ec05d283c25586c3c8caf7487af583d70821" Nov 03 20:01:18.318222 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:18.301177 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:01:18.318222 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:18.301339 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:01:18.318222 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:18.301415 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:01:19.184566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:19.183872 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:01:19.184566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:19.184001 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:01:19.184566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:19.184044 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" podUID="1e1df657-2ab6-4ae7-80e5-0c6358becefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 20:01:19.310446 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:19.309822 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" event={"ID":"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c","Type":"ContainerStarted","Data":"d093afa4d10294a4d4661b5d96263fc434fd51f7efeb98e86cc285976a039859"} Nov 03 20:01:19.355558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:19.355053 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:01:19.355558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:19.355187 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:01:19.355558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:19.355236 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" podUID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 20:01:20.174553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.173974 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:01:20.174553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.174053 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" podUID="1e1df657-2ab6-4ae7-80e5-0c6358becefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 20:01:20.363557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.355983 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:01:20.363557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.356050 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" podUID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 20:01:20.838689 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.838654 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm"] Nov 03 20:01:20.843128 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.842763 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:20.903618 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.900232 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-11a5b0\"/\"default-dockercfg-rm8q5\"" Nov 03 20:01:20.903618 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.900325 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-11a5b0\"/\"openshift-service-ca.crt\"" Nov 03 20:01:20.903618 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.900392 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-11a5b0\"/\"kube-root-ca.crt\"" Nov 03 20:01:20.903618 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.901196 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-11a5b0\"/\"simulator-w2-11a5b0-kserve-self-signed-certs\"" Nov 03 20:01:20.911822 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.910353 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm"] Nov 03 20:01:20.931103 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.931062 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-dshm\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:20.931282 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.931154 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab2b92fb-74eb-42cc-a77a-26871be8a997-tls-certs\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:20.931282 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.931205 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-model-cache\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:20.931415 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.931313 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-tmp-dir\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:20.931415 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.931340 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-home\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:20.931515 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:20.931488 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nm6c\" (UniqueName: \"kubernetes.io/projected/ab2b92fb-74eb-42cc-a77a-26871be8a997-kube-api-access-8nm6c\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:21.034264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.032979 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nm6c\" (UniqueName: \"kubernetes.io/projected/ab2b92fb-74eb-42cc-a77a-26871be8a997-kube-api-access-8nm6c\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:21.034264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.033062 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-dshm\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:21.034264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.033124 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab2b92fb-74eb-42cc-a77a-26871be8a997-tls-certs\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:21.034264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.033174 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-model-cache\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:21.034264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.033220 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-tmp-dir\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:21.034264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.033247 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-home\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:21.034264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.033765 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-home\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:21.036821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.034893 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-tmp-dir\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:21.036821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.035073 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-model-cache\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:21.038777 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.038752 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-dshm\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:21.039162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.039140 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab2b92fb-74eb-42cc-a77a-26871be8a997-tls-certs\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:21.112600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.110969 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nm6c\" (UniqueName: \"kubernetes.io/projected/ab2b92fb-74eb-42cc-a77a-26871be8a997-kube-api-access-8nm6c\") pod \"simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:21.174556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.173962 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:01:21.174556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.174025 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" podUID="1e1df657-2ab6-4ae7-80e5-0c6358becefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 20:01:21.222770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.222457 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:21.358514 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.353926 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:01:21.358514 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.353993 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" podUID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 20:01:21.698114 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:21.696351 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm"] Nov 03 20:01:22.180558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:22.179948 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:01:22.180558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:22.180014 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" podUID="1e1df657-2ab6-4ae7-80e5-0c6358becefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 20:01:22.339158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:22.338572 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" event={"ID":"ab2b92fb-74eb-42cc-a77a-26871be8a997","Type":"ContainerStarted","Data":"556af6cb6da2f941a88c53d85d342ad9d80966eb98c617038faa48ae2e393fc1"} Nov 03 20:01:22.339158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:22.338619 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" event={"ID":"ab2b92fb-74eb-42cc-a77a-26871be8a997","Type":"ContainerStarted","Data":"3eab62860a4209d12b9653fd74145840c888abdd155e0101fb0fd5d85c78bd3e"} Nov 03 20:01:22.354008 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:22.353966 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:01:22.354314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:22.354283 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" podUID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 20:01:22.395275 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:22.395201 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podStartSLOduration=2.395175355 podStartE2EDuration="2.395175355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:22.386990094 +0000 UTC m=+1312.349368570" watchObservedRunningTime="2026-11-03 20:01:22.395175355 +0000 UTC m=+1312.357553796" Nov 03 20:01:23.176976 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:23.176940 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 20:01:23.177740 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:23.177700 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" podUID="1e1df657-2ab6-4ae7-80e5-0c6358becefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 20:01:23.223685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:23.223650 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:23.354566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:23.352914 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:01:23.354566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:23.352977 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" podUID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 20:01:23.628936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:23.628890 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:23.643149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:23.643118 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:23.928805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:23.928715 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:01:23.928805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:23.928778 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:01:24.172107 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:24.171667 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": read tcp 10.132.0.2:43924->10.132.0.61:15021: read: connection reset by peer" start-of-body= Nov 03 20:01:24.172107 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:24.171732 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" podUID="97388cfd-a7f0-425d-946b-bf0181f548b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": read tcp 10.132.0.2:43924->10.132.0.61:15021: read: connection reset by peer" Nov 03 20:01:24.187078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:24.186908 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:01:24.370330 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:24.370281 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:01:24.370569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:24.370364 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" podUID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 20:01:24.376491 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:24.376466 5111 generic.go:356] "Generic (PLEG): container finished" podID="97388cfd-a7f0-425d-946b-bf0181f548b0" containerID="7066ba48898c82f9859b2cf5b411008139e2782c58d7ee14dc575085b7bcee1c" exitCode=0 Nov 03 20:01:24.376671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:24.376584 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" event={"ID":"97388cfd-a7f0-425d-946b-bf0181f548b0","Type":"ContainerDied","Data":"7066ba48898c82f9859b2cf5b411008139e2782c58d7ee14dc575085b7bcee1c"} Nov 03 20:01:24.377039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:24.377019 5111 scope.go:122] "RemoveContainer" containerID="7066ba48898c82f9859b2cf5b411008139e2782c58d7ee14dc575085b7bcee1c" Nov 03 20:01:24.378218 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:24.377817 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:01:24.378684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:24.378660 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:01:24.378764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:24.378751 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:01:24.378802 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:24.378789 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:01:24.392713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:24.392683 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:01:25.320276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.320236 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446"] Nov 03 20:01:25.323310 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.323288 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.328993 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.328954 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7959c647-ef80-40d4-8262-335b9e72934b-tls-certs\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.329162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.329028 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-dshm\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.329162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.329090 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-model-cache\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.329162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.329128 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-home\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.329327 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.329202 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-tmp-dir\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.329327 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.329248 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vrqm\" (UniqueName: \"kubernetes.io/projected/7959c647-ef80-40d4-8262-335b9e72934b-kube-api-access-9vrqm\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.334113 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.334087 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b42aea\"/\"simulator-w1-b42aea-kserve-self-signed-certs\"" Nov 03 20:01:25.334284 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.334170 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b42aea\"/\"openshift-service-ca.crt\"" Nov 03 20:01:25.334366 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.334176 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b42aea\"/\"kube-root-ca.crt\"" Nov 03 20:01:25.338115 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.338085 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446"] Nov 03 20:01:25.340329 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.340309 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b42aea\"/\"default-dockercfg-8tctz\"" Nov 03 20:01:25.382092 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.382063 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:01:25.383619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.383593 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:01:25.391614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.391579 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" event={"ID":"97388cfd-a7f0-425d-946b-bf0181f548b0","Type":"ContainerStarted","Data":"49a572d4425500309fe26eac2e1d22038f7c103faee0d2fce8204dec65f313f5"} Nov 03 20:01:25.431409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.431355 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7959c647-ef80-40d4-8262-335b9e72934b-tls-certs\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.431632 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.431522 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-dshm\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.431713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.431633 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-model-cache\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.431713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.431673 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-home\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.431874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.431852 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-tmp-dir\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.431955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.431934 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vrqm\" (UniqueName: \"kubernetes.io/projected/7959c647-ef80-40d4-8262-335b9e72934b-kube-api-access-9vrqm\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.433425 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.433293 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-home\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.433614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.433588 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-tmp-dir\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.433888 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.433855 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-model-cache\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.439102 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.439063 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-dshm\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.439506 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.439480 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7959c647-ef80-40d4-8262-335b9e72934b-tls-certs\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.479200 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.452800 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vrqm\" (UniqueName: \"kubernetes.io/projected/7959c647-ef80-40d4-8262-335b9e72934b-kube-api-access-9vrqm\") pod \"simulator-w1-b42aea-kserve-7f68f546b7-s7446\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.637403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.637362 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:25.971429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:25.971194 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446"] Nov 03 20:01:26.100150 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:26.100117 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:01:26.100431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:26.100399 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:01:26.100566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:26.100459 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" podUID="97388cfd-a7f0-425d-946b-bf0181f548b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 20:01:26.398138 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:26.398094 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" event={"ID":"7959c647-ef80-40d4-8262-335b9e72934b","Type":"ContainerStarted","Data":"165652cafb9f442391bee353f07035570477f7d2c9ac2f7c114ad616eedfd14b"} Nov 03 20:01:26.398138 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:26.398140 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" event={"ID":"7959c647-ef80-40d4-8262-335b9e72934b","Type":"ContainerStarted","Data":"b5fd76c8245305e9146acca9bc512ca361e68624763f109d41ac8dde9ce7d73a"} Nov 03 20:01:26.419615 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:26.419551 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podStartSLOduration=1.4195247659999999 podStartE2EDuration="1.419524766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:26.416613805 +0000 UTC m=+1316.378992243" watchObservedRunningTime="2026-11-03 20:01:26.419524766 +0000 UTC m=+1316.381903204" Nov 03 20:01:26.638559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:26.638490 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:27.100011 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:27.099975 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:01:27.100206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:27.100029 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" podUID="97388cfd-a7f0-425d-946b-bf0181f548b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 20:01:28.100575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:28.100523 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:01:28.101026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:28.100608 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" podUID="97388cfd-a7f0-425d-946b-bf0181f548b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 20:01:29.100522 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:29.100482 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:01:29.100852 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:29.100820 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" podUID="97388cfd-a7f0-425d-946b-bf0181f548b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 20:01:30.099905 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:30.099867 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:01:30.100134 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:30.099928 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" podUID="97388cfd-a7f0-425d-946b-bf0181f548b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 20:01:31.223195 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.223150 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:31.235679 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.235639 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:31.297078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.297041 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:01:31.297495 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.297466 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:01:31.298276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.298256 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:01:31.445260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.445218 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:01:31.496762 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.496667 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg"] Nov 03 20:01:31.500352 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.500303 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.505723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.505691 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-0ed729\"/\"simulator-w6-0ed729-kserve-self-signed-certs\"" Nov 03 20:01:31.508803 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.508775 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-0ed729\"/\"default-dockercfg-pmtnk\"" Nov 03 20:01:31.513562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.513525 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-0ed729\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.515164 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.513823 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-0ed729\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.525112 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.525081 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg"] Nov 03 20:01:31.539202 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.539160 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gjh9\" (UniqueName: \"kubernetes.io/projected/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-kube-api-access-5gjh9\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.539372 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.539343 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-home\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.539442 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.539401 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-tmp-dir\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.539506 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.539459 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-dshm\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.539589 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.539570 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-tls-certs\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.539656 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.539614 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-model-cache\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.641251 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.641217 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-home\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.641448 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.641267 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-tmp-dir\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.641490 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.641467 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-dshm\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.641591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.641561 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-tls-certs\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.641650 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.641605 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-model-cache\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.641744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.641710 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gjh9\" (UniqueName: \"kubernetes.io/projected/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-kube-api-access-5gjh9\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.641744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.641737 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-home\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.641945 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.641761 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-tmp-dir\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.642016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.641983 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-model-cache\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.643754 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.643726 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-dshm\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.644228 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.644207 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-tls-certs\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.658922 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.658895 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gjh9\" (UniqueName: \"kubernetes.io/projected/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-kube-api-access-5gjh9\") pod \"simulator-w6-0ed729-kserve-7b95b48f76-9d4tg\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:31.819605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:31.819506 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:32.095928 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:32.095896 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg"] Nov 03 20:01:32.097653 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:01:32.097619 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bd3b520_ce5d_4bfd_9fcf_89b385318ad8.slice/crio-a01b53550954fdf4c9031401077ea07cd0e1459c7514c9d3f163f8c426bdb68f WatchSource:0}: Error finding container a01b53550954fdf4c9031401077ea07cd0e1459c7514c9d3f163f8c426bdb68f: Status 404 returned error can't find the container with id a01b53550954fdf4c9031401077ea07cd0e1459c7514c9d3f163f8c426bdb68f Nov 03 20:01:32.448342 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:32.448305 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" event={"ID":"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8","Type":"ContainerStarted","Data":"a76d15f0aa5639d6ad292234693d08d5e96a18dce9d700d46126dd62f10fad9b"} Nov 03 20:01:32.448835 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:32.448352 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" event={"ID":"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8","Type":"ContainerStarted","Data":"a01b53550954fdf4c9031401077ea07cd0e1459c7514c9d3f163f8c426bdb68f"} Nov 03 20:01:32.468589 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:32.468525 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podStartSLOduration=1.468510481 podStartE2EDuration="1.468510481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:32.466784712 +0000 UTC m=+1322.429163149" watchObservedRunningTime="2026-11-03 20:01:32.468510481 +0000 UTC m=+1322.430888919" Nov 03 20:01:32.820266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:32.820177 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:35.060950 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:01:35.060903 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:01:35.638854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:35.638814 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:35.652032 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:35.652000 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:36.482972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:36.482941 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:01:41.820744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:41.820704 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:41.834389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:41.834355 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:42.521829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:42.521791 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:01:45.958304 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:45.958265 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks"] Nov 03 20:01:45.962584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:45.962557 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:45.972249 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:45.972220 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks"] Nov 03 20:01:45.975166 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:45.975142 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-11a5b0\"/\"premium-w2-11a5b0-kserve-self-signed-certs\"" Nov 03 20:01:46.124000 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.123963 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-dshm\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.124321 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.124296 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f87jf\" (UniqueName: \"kubernetes.io/projected/275b0913-a9c9-4165-b1b1-be6e85c7294a-kube-api-access-f87jf\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.124484 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.124458 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-home\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.124735 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.124599 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-model-cache\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.124897 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.124877 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275b0913-a9c9-4165-b1b1-be6e85c7294a-tls-certs\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.125289 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.125255 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-tmp-dir\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.226144 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.226060 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-dshm\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.226144 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.226114 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87jf\" (UniqueName: \"kubernetes.io/projected/275b0913-a9c9-4165-b1b1-be6e85c7294a-kube-api-access-f87jf\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.226332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.226147 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-home\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.226332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.226199 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-model-cache\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.226332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.226320 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275b0913-a9c9-4165-b1b1-be6e85c7294a-tls-certs\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.226469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.226389 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-tmp-dir\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.226612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.226587 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-model-cache\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.226743 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.226646 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-home\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.226848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.226824 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-tmp-dir\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.228431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.228411 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-dshm\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.228806 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.228789 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275b0913-a9c9-4165-b1b1-be6e85c7294a-tls-certs\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.233802 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.233781 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f87jf\" (UniqueName: \"kubernetes.io/projected/275b0913-a9c9-4165-b1b1-be6e85c7294a-kube-api-access-f87jf\") pod \"premium-w2-11a5b0-kserve-5ff7bc548b-fdcks\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.279934 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.279895 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:46.560856 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:46.560780 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks"] Nov 03 20:01:46.564176 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:01:46.564137 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod275b0913_a9c9_4165_b1b1_be6e85c7294a.slice/crio-07513c9fe48f549e263a465a893fdf79fa44b4252d7a0336fffae2d394b7ec81 WatchSource:0}: Error finding container 07513c9fe48f549e263a465a893fdf79fa44b4252d7a0336fffae2d394b7ec81: Status 404 returned error can't find the container with id 07513c9fe48f549e263a465a893fdf79fa44b4252d7a0336fffae2d394b7ec81 Nov 03 20:01:47.552922 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:47.552888 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" event={"ID":"275b0913-a9c9-4165-b1b1-be6e85c7294a","Type":"ContainerStarted","Data":"1fbc34268ceaa29c2fbf32e4e27da8be4c99d43e1d94bc361cd1808d60dad915"} Nov 03 20:01:47.553378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:47.552931 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" event={"ID":"275b0913-a9c9-4165-b1b1-be6e85c7294a","Type":"ContainerStarted","Data":"07513c9fe48f549e263a465a893fdf79fa44b4252d7a0336fffae2d394b7ec81"} Nov 03 20:01:47.572326 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:47.572270 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podStartSLOduration=2.572256112 podStartE2EDuration="2.572256112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:47.569760934 +0000 UTC m=+1337.532139374" watchObservedRunningTime="2026-11-03 20:01:47.572256112 +0000 UTC m=+1337.534634551" Nov 03 20:01:48.280677 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:48.280632 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:54.050557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.050495 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l"] Nov 03 20:01:54.056804 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.056775 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l"] Nov 03 20:01:54.056959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.056875 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.064041 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.064007 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b42aea\"/\"premium-w1-b42aea-kserve-self-signed-certs\"" Nov 03 20:01:54.151106 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.151064 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjpww\" (UniqueName: \"kubernetes.io/projected/ec0ff9ad-04a1-4709-960f-78e78bbaca58-kube-api-access-tjpww\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.151106 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.151104 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0ff9ad-04a1-4709-960f-78e78bbaca58-tls-certs\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.151377 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.151129 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-tmp-dir\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.151377 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.151258 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-home\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.151377 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.151274 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-dshm\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.151377 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.151320 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-model-cache\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.252283 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.252238 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-model-cache\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.252456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.252347 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjpww\" (UniqueName: \"kubernetes.io/projected/ec0ff9ad-04a1-4709-960f-78e78bbaca58-kube-api-access-tjpww\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.252456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.252367 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0ff9ad-04a1-4709-960f-78e78bbaca58-tls-certs\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.252456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.252390 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-tmp-dir\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.252456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.252444 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-home\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.252723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.252464 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-dshm\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.252723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.252704 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-model-cache\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.252856 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.252822 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-home\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.252899 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.252850 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-tmp-dir\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.254817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.254791 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-dshm\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.255092 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.255075 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0ff9ad-04a1-4709-960f-78e78bbaca58-tls-certs\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.273543 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.273491 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjpww\" (UniqueName: \"kubernetes.io/projected/ec0ff9ad-04a1-4709-960f-78e78bbaca58-kube-api-access-tjpww\") pod \"premium-w1-b42aea-kserve-5d994f8d94-tqg7l\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.375498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.375466 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:54.655746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:54.655716 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l"] Nov 03 20:01:54.657766 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:01:54.657733 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec0ff9ad_04a1_4709_960f_78e78bbaca58.slice/crio-5222a82f347a966efce5d28fc270f6eb63f3306ead1924c3f53c3a36e8eb43d3 WatchSource:0}: Error finding container 5222a82f347a966efce5d28fc270f6eb63f3306ead1924c3f53c3a36e8eb43d3: Status 404 returned error can't find the container with id 5222a82f347a966efce5d28fc270f6eb63f3306ead1924c3f53c3a36e8eb43d3 Nov 03 20:01:55.604704 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:55.604667 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" event={"ID":"ec0ff9ad-04a1-4709-960f-78e78bbaca58","Type":"ContainerStarted","Data":"9795e1c7a47dd18c7b53e7fe23b8e952d5a0bc8c0c98a2a580e4e4264edb52cd"} Nov 03 20:01:55.605236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:55.604714 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" event={"ID":"ec0ff9ad-04a1-4709-960f-78e78bbaca58","Type":"ContainerStarted","Data":"5222a82f347a966efce5d28fc270f6eb63f3306ead1924c3f53c3a36e8eb43d3"} Nov 03 20:01:55.622613 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:55.622552 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podStartSLOduration=2.622526346 podStartE2EDuration="2.622526346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:55.621124979 +0000 UTC m=+1345.583503419" watchObservedRunningTime="2026-11-03 20:01:55.622526346 +0000 UTC m=+1345.584904785" Nov 03 20:01:56.280385 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:56.280343 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:56.293551 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:56.293508 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:01:56.376491 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:56.376452 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:01:56.629297 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:01:56.629260 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:02:04.122591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.122548 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784"] Nov 03 20:02:04.128138 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.128109 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.136604 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.136574 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784"] Nov 03 20:02:04.136935 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.136909 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b87733\"/\"openshift-service-ca.crt\"" Nov 03 20:02:04.137787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.137204 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b87733\"/\"premium-w3-b87733-kserve-self-signed-certs\"" Nov 03 20:02:04.138593 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.138573 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b87733\"/\"kube-root-ca.crt\"" Nov 03 20:02:04.138732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.138676 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b87733\"/\"default-dockercfg-9qptp\"" Nov 03 20:02:04.183698 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.183666 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-dshm\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.183873 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.183736 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-model-cache\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.183873 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.183816 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xs79\" (UniqueName: \"kubernetes.io/projected/5333c660-c55d-40f0-b0cc-8decef5eeb22-kube-api-access-9xs79\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.183873 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.183859 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-tmp-dir\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.183989 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.183968 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5333c660-c55d-40f0-b0cc-8decef5eeb22-tls-certs\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.184027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.184001 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-home\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.285123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.285080 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-home\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.285436 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.285187 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-dshm\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.285436 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.285296 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-model-cache\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.285436 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.285384 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xs79\" (UniqueName: \"kubernetes.io/projected/5333c660-c55d-40f0-b0cc-8decef5eeb22-kube-api-access-9xs79\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.285639 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.285437 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-tmp-dir\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.285639 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.285567 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5333c660-c55d-40f0-b0cc-8decef5eeb22-tls-certs\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.286282 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.286255 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-model-cache\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.289020 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.286715 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-home\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.289020 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.286900 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-tmp-dir\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.289453 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.289429 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-dshm\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.290122 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.290099 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5333c660-c55d-40f0-b0cc-8decef5eeb22-tls-certs\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.297862 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.297824 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xs79\" (UniqueName: \"kubernetes.io/projected/5333c660-c55d-40f0-b0cc-8decef5eeb22-kube-api-access-9xs79\") pod \"premium-w3-b87733-kserve-54b489bc6f-bk784\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.376736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.376649 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:02:04.390173 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.390141 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:02:04.443497 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.443455 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:04.679293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.679208 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:02:04.734301 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:04.734268 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784"] Nov 03 20:02:04.736139 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:02:04.736085 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5333c660_c55d_40f0_b0cc_8decef5eeb22.slice/crio-51a4b73281b1d5ff3a82348167bc2d647fe6c220737e2bbb759d36b62e5a5fa0 WatchSource:0}: Error finding container 51a4b73281b1d5ff3a82348167bc2d647fe6c220737e2bbb759d36b62e5a5fa0: Status 404 returned error can't find the container with id 51a4b73281b1d5ff3a82348167bc2d647fe6c220737e2bbb759d36b62e5a5fa0 Nov 03 20:02:05.064476 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:02:05.064392 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:02:05.675188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:05.675152 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" event={"ID":"5333c660-c55d-40f0-b0cc-8decef5eeb22","Type":"ContainerStarted","Data":"e1403371711eec094adf68f6d9c62e935510394c6bb9b4e37d522b12c7c1ba64"} Nov 03 20:02:05.675188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:05.675200 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" event={"ID":"5333c660-c55d-40f0-b0cc-8decef5eeb22","Type":"ContainerStarted","Data":"51a4b73281b1d5ff3a82348167bc2d647fe6c220737e2bbb759d36b62e5a5fa0"} Nov 03 20:02:05.693854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:05.693808 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podStartSLOduration=1.693795857 podStartE2EDuration="1.693795857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:05.692074809 +0000 UTC m=+1355.654453242" watchObservedRunningTime="2026-11-03 20:02:05.693795857 +0000 UTC m=+1355.656174295" Nov 03 20:02:06.444204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:06.444170 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:07.292452 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.292412 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65"] Nov 03 20:02:07.297667 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.297647 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.305370 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.305343 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ab1e46-openshift-default-dockercfg-v2sht\"" Nov 03 20:02:07.310796 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.310766 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65"] Nov 03 20:02:07.429425 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.429381 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.429696 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.429481 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.429696 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.429550 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.429696 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.429588 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.429696 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.429630 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.429696 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.429658 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590bd9c8-3170-40be-89b4-15d8df5acd02-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.429991 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.429748 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.429991 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.429795 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.429991 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.429835 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s868\" (UniqueName: \"kubernetes.io/projected/590bd9c8-3170-40be-89b4-15d8df5acd02-kube-api-access-2s868\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.531565 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.531509 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.531785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.531587 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.531785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.531627 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.531785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.531654 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590bd9c8-3170-40be-89b4-15d8df5acd02-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.531785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.531734 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.531785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.531781 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.532074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.531819 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s868\" (UniqueName: \"kubernetes.io/projected/590bd9c8-3170-40be-89b4-15d8df5acd02-kube-api-access-2s868\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.532074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.531888 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.532074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.531959 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.532431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.532405 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.532527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.532480 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.532622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.532526 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.532764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.532746 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590bd9c8-3170-40be-89b4-15d8df5acd02-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.532857 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.532832 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.534422 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.534394 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.534656 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.534638 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.543868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.543809 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.544125 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.544107 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s868\" (UniqueName: \"kubernetes.io/projected/590bd9c8-3170-40be-89b4-15d8df5acd02-kube-api-access-2s868\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.614132 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.614096 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:07.906894 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.906866 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65"] Nov 03 20:02:07.908750 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:02:07.908722 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590bd9c8_3170_40be_89b4_15d8df5acd02.slice/crio-8a439770db976d0acb5db0abeab64ff6eb8704b54e468f70dd4dd435952fad53 WatchSource:0}: Error finding container 8a439770db976d0acb5db0abeab64ff6eb8704b54e468f70dd4dd435952fad53: Status 404 returned error can't find the container with id 8a439770db976d0acb5db0abeab64ff6eb8704b54e468f70dd4dd435952fad53 Nov 03 20:02:07.911426 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.911389 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:02:07.911574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.911471 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:02:07.911574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:07.911516 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:02:08.712062 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:08.712015 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" event={"ID":"590bd9c8-3170-40be-89b4-15d8df5acd02","Type":"ContainerStarted","Data":"8356ad6cb6fa76a741dc59a31ea18a1f3f2c1041416517d93621d6dd8cf8a7fe"} Nov 03 20:02:08.712607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:08.712117 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" event={"ID":"590bd9c8-3170-40be-89b4-15d8df5acd02","Type":"ContainerStarted","Data":"8a439770db976d0acb5db0abeab64ff6eb8704b54e468f70dd4dd435952fad53"} Nov 03 20:02:08.734613 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:08.734564 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" podStartSLOduration=1.734547986 podStartE2EDuration="1.734547986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:08.731923641 +0000 UTC m=+1358.694302092" watchObservedRunningTime="2026-11-03 20:02:08.734547986 +0000 UTC m=+1358.696926419" Nov 03 20:02:09.615059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:09.615021 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:09.617050 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:09.617021 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:02:09.617212 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:09.617069 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" podUID="590bd9c8-3170-40be-89b4-15d8df5acd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:02:10.615426 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:10.615388 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:02:10.615869 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:10.615455 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" podUID="590bd9c8-3170-40be-89b4-15d8df5acd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:02:11.615094 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:11.615061 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:02:11.615274 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:11.615107 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" podUID="590bd9c8-3170-40be-89b4-15d8df5acd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:02:12.614481 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:12.614445 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:02:12.614944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:12.614504 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" podUID="590bd9c8-3170-40be-89b4-15d8df5acd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:02:13.615394 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:13.615360 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:02:13.615830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:13.615419 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" podUID="590bd9c8-3170-40be-89b4-15d8df5acd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:02:14.444329 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:14.444290 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:14.457213 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:14.457183 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:14.761803 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:14.761719 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:02:14.819191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:14.819138 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:14.820692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:14.820669 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:14.826862 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:14.826839 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:02:35.057071 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:02:35.057031 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:02:37.146579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.146510 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w"] Nov 03 20:02:37.152823 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.152797 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.169033 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.168986 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b87733-openshift-default-dockercfg-lvxzx\"" Nov 03 20:02:37.173823 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.173796 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w"] Nov 03 20:02:37.204032 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.203984 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf7d094-ba6e-4c14-b731-c13068585df1-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.204215 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.204072 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.204215 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.204128 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.204215 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.204160 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.204406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.204238 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.204406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.204282 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.204406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.204324 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.204406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.204380 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4n2f\" (UniqueName: \"kubernetes.io/projected/8bf7d094-ba6e-4c14-b731-c13068585df1-kube-api-access-n4n2f\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.204681 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.204419 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.304384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.304344 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv"] Nov 03 20:02:37.306294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.306245 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf7d094-ba6e-4c14-b731-c13068585df1-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.306430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.306355 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.306430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.306405 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.306714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.306691 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.306817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.306801 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.306878 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.306853 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.306933 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.306899 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.306986 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.306959 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4n2f\" (UniqueName: \"kubernetes.io/projected/8bf7d094-ba6e-4c14-b731-c13068585df1-kube-api-access-n4n2f\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.307043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.306998 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.307096 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.307041 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.307867 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.307212 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf7d094-ba6e-4c14-b731-c13068585df1-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.307867 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.307368 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.307867 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.307824 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.307867 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.307854 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.309308 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.309275 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.309849 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.309820 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.310892 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.310870 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.319631 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.319606 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv"] Nov 03 20:02:37.328224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.328200 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4n2f\" (UniqueName: \"kubernetes.io/projected/8bf7d094-ba6e-4c14-b731-c13068585df1-kube-api-access-n4n2f\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.329141 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.329116 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.365479 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.365447 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-0ed729-openshift-default-dockercfg-cflmg\"" Nov 03 20:02:37.416598 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.416484 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.416598 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.416543 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.416819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.416599 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.416819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.416663 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pthhg\" (UniqueName: \"kubernetes.io/projected/7310a5cf-4706-445f-8c50-95e2f57c1ee2-kube-api-access-pthhg\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.416819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.416717 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.416819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.416751 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.416819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.416813 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.417073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.416874 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.417073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.416931 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.485070 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.485032 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:37.520769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.518251 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.520769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.518315 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.520769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.518377 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.520769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.518432 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.520769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.518490 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.520769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.518611 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.520769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.518641 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.520769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.518696 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.520769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.518770 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pthhg\" (UniqueName: \"kubernetes.io/projected/7310a5cf-4706-445f-8c50-95e2f57c1ee2-kube-api-access-pthhg\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.520769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.519217 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.520769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.519546 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.520769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.519641 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.520769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.519790 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.520769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.519838 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.522734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.522709 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.523780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.523758 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.543139 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.543103 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.545230 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.545200 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pthhg\" (UniqueName: \"kubernetes.io/projected/7310a5cf-4706-445f-8c50-95e2f57c1ee2-kube-api-access-pthhg\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.673431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.673341 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:37.800474 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:02:37.800437 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf7d094_ba6e_4c14_b731_c13068585df1.slice/crio-af3664d2891452f2be9951b2ae49a48d2256b3a55934f2aad36ac64dbca96c8c WatchSource:0}: Error finding container af3664d2891452f2be9951b2ae49a48d2256b3a55934f2aad36ac64dbca96c8c: Status 404 returned error can't find the container with id af3664d2891452f2be9951b2ae49a48d2256b3a55934f2aad36ac64dbca96c8c Nov 03 20:02:37.801023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.800991 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w"] Nov 03 20:02:37.803802 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.803763 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:02:37.803913 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.803844 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:02:37.803913 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.803907 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:02:37.920187 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.920149 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" event={"ID":"8bf7d094-ba6e-4c14-b731-c13068585df1","Type":"ContainerStarted","Data":"d0a9bbe12d9d63b469c03867a5799429153313f12e9ab36c577f56a29f4724fc"} Nov 03 20:02:37.920369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.920217 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" event={"ID":"8bf7d094-ba6e-4c14-b731-c13068585df1","Type":"ContainerStarted","Data":"af3664d2891452f2be9951b2ae49a48d2256b3a55934f2aad36ac64dbca96c8c"} Nov 03 20:02:37.927696 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.927626 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg"] Nov 03 20:02:37.945358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.945314 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" Nov 03 20:02:37.960995 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.960929 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:02:37.961463 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.961443 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:02:37.961749 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.961733 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:02:37.961967 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.961951 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:02:37.977451 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.977432 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg"] Nov 03 20:02:37.977970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:37.977915 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" podStartSLOduration=0.97789951 podStartE2EDuration="977.89951ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:37.955801398 +0000 UTC m=+1387.918179839" watchObservedRunningTime="2026-11-03 20:02:37.97789951 +0000 UTC m=+1387.940277953" Nov 03 20:02:38.020594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.017944 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv"] Nov 03 20:02:38.022208 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:02:38.022178 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7310a5cf_4706_445f_8c50_95e2f57c1ee2.slice/crio-4e1c7324b7c82a55c0c262a128898780eeb82f9cd9ed91ffece3d0f9b10d8bfb WatchSource:0}: Error finding container 4e1c7324b7c82a55c0c262a128898780eeb82f9cd9ed91ffece3d0f9b10d8bfb: Status 404 returned error can't find the container with id 4e1c7324b7c82a55c0c262a128898780eeb82f9cd9ed91ffece3d0f9b10d8bfb Nov 03 20:02:38.025115 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.025071 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:02:38.025236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.025168 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:02:38.025236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.025213 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:02:38.028066 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.028037 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9423b69f-7ef4-43ed-813e-473355ff28c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b0f5cc3-zzdjg\" (UID: \"9423b69f-7ef4-43ed-813e-473355ff28c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" Nov 03 20:02:38.028382 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.028257 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb8qc\" (UniqueName: \"kubernetes.io/projected/9423b69f-7ef4-43ed-813e-473355ff28c7-kube-api-access-zb8qc\") pod \"maas-api-revoke-sub-1b0f5cc3-zzdjg\" (UID: \"9423b69f-7ef4-43ed-813e-473355ff28c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" Nov 03 20:02:38.028495 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.028395 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9423b69f-7ef4-43ed-813e-473355ff28c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b0f5cc3-zzdjg\" (UID: \"9423b69f-7ef4-43ed-813e-473355ff28c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" Nov 03 20:02:38.129499 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.129469 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb8qc\" (UniqueName: \"kubernetes.io/projected/9423b69f-7ef4-43ed-813e-473355ff28c7-kube-api-access-zb8qc\") pod \"maas-api-revoke-sub-1b0f5cc3-zzdjg\" (UID: \"9423b69f-7ef4-43ed-813e-473355ff28c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" Nov 03 20:02:38.129748 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.129623 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9423b69f-7ef4-43ed-813e-473355ff28c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b0f5cc3-zzdjg\" (UID: \"9423b69f-7ef4-43ed-813e-473355ff28c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" Nov 03 20:02:38.129818 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.129746 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9423b69f-7ef4-43ed-813e-473355ff28c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b0f5cc3-zzdjg\" (UID: \"9423b69f-7ef4-43ed-813e-473355ff28c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" Nov 03 20:02:38.130832 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.130806 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9423b69f-7ef4-43ed-813e-473355ff28c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b0f5cc3-zzdjg\" (UID: \"9423b69f-7ef4-43ed-813e-473355ff28c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" Nov 03 20:02:38.132999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.132699 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9423b69f-7ef4-43ed-813e-473355ff28c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b0f5cc3-zzdjg\" (UID: \"9423b69f-7ef4-43ed-813e-473355ff28c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" Nov 03 20:02:38.139184 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.139151 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb8qc\" (UniqueName: \"kubernetes.io/projected/9423b69f-7ef4-43ed-813e-473355ff28c7-kube-api-access-zb8qc\") pod \"maas-api-revoke-sub-1b0f5cc3-zzdjg\" (UID: \"9423b69f-7ef4-43ed-813e-473355ff28c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" Nov 03 20:02:38.288958 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.288879 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" Nov 03 20:02:38.489577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.487995 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:38.491660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.491623 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:02:38.491824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.491682 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" podUID="8bf7d094-ba6e-4c14-b731-c13068585df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:02:38.619741 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.619673 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg"] Nov 03 20:02:38.642063 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:02:38.642030 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9423b69f_7ef4_43ed_813e_473355ff28c7.slice/crio-05d6f04d1a5d0635d55e1fee1d6d9a2ef1ae9da2290d2ce25394829271210257 WatchSource:0}: Error finding container 05d6f04d1a5d0635d55e1fee1d6d9a2ef1ae9da2290d2ce25394829271210257: Status 404 returned error can't find the container with id 05d6f04d1a5d0635d55e1fee1d6d9a2ef1ae9da2290d2ce25394829271210257 Nov 03 20:02:38.929406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.929301 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" event={"ID":"7310a5cf-4706-445f-8c50-95e2f57c1ee2","Type":"ContainerStarted","Data":"ff970f9879c31eda0bd975b875951654bf74d5231a34d00487b28714979e50fb"} Nov 03 20:02:38.929406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.929355 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" event={"ID":"7310a5cf-4706-445f-8c50-95e2f57c1ee2","Type":"ContainerStarted","Data":"4e1c7324b7c82a55c0c262a128898780eeb82f9cd9ed91ffece3d0f9b10d8bfb"} Nov 03 20:02:38.931459 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.930797 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" event={"ID":"9423b69f-7ef4-43ed-813e-473355ff28c7","Type":"ContainerStarted","Data":"05d6f04d1a5d0635d55e1fee1d6d9a2ef1ae9da2290d2ce25394829271210257"} Nov 03 20:02:38.969128 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:38.965324 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" podStartSLOduration=1.965289016 podStartE2EDuration="1.965289016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:38.961872456 +0000 UTC m=+1388.924250896" watchObservedRunningTime="2026-11-03 20:02:38.965289016 +0000 UTC m=+1388.927667456" Nov 03 20:02:39.490957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:39.489894 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:02:39.490957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:39.489957 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" podUID="8bf7d094-ba6e-4c14-b731-c13068585df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:02:39.673720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:39.673687 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:39.682669 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:39.682635 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:02:39.682910 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:39.682880 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" podUID="7310a5cf-4706-445f-8c50-95e2f57c1ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 20:02:39.944559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:39.942437 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" event={"ID":"9423b69f-7ef4-43ed-813e-473355ff28c7","Type":"ContainerStarted","Data":"4da38a879655be99d50e7f4d7c779ed486539dacfb84e8567961b0bef8c40a08"} Nov 03 20:02:39.972554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:39.970795 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" podStartSLOduration=2.970774328 podStartE2EDuration="2.970774328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:39.969875844 +0000 UTC m=+1389.932254285" watchObservedRunningTime="2026-11-03 20:02:39.970774328 +0000 UTC m=+1389.933152768" Nov 03 20:02:40.487556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:40.486905 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:02:40.487556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:40.486968 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" podUID="8bf7d094-ba6e-4c14-b731-c13068585df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:02:40.678555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:40.677469 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:02:40.678555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:40.677558 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" podUID="7310a5cf-4706-445f-8c50-95e2f57c1ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 20:02:40.970562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:40.969348 5111 generic.go:356] "Generic (PLEG): container finished" podID="9423b69f-7ef4-43ed-813e-473355ff28c7" containerID="4da38a879655be99d50e7f4d7c779ed486539dacfb84e8567961b0bef8c40a08" exitCode=0 Nov 03 20:02:40.970562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:40.969950 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" event={"ID":"9423b69f-7ef4-43ed-813e-473355ff28c7","Type":"ContainerDied","Data":"4da38a879655be99d50e7f4d7c779ed486539dacfb84e8567961b0bef8c40a08"} Nov 03 20:02:41.494559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:41.486352 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:02:41.494559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:41.486417 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" podUID="8bf7d094-ba6e-4c14-b731-c13068585df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:02:41.681557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:41.680868 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:02:41.681557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:41.680943 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" podUID="7310a5cf-4706-445f-8c50-95e2f57c1ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 20:02:42.306097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:42.306067 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" Nov 03 20:02:42.401578 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:42.375022 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9423b69f-7ef4-43ed-813e-473355ff28c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9423b69f-7ef4-43ed-813e-473355ff28c7-maas-api-xks-ca\") pod \"9423b69f-7ef4-43ed-813e-473355ff28c7\" (UID: \"9423b69f-7ef4-43ed-813e-473355ff28c7\") " Nov 03 20:02:42.401578 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:42.375193 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9423b69f-7ef4-43ed-813e-473355ff28c7-kube-api-access-zb8qc\" (UniqueName: \"kubernetes.io/projected/9423b69f-7ef4-43ed-813e-473355ff28c7-kube-api-access-zb8qc\") pod \"9423b69f-7ef4-43ed-813e-473355ff28c7\" (UID: \"9423b69f-7ef4-43ed-813e-473355ff28c7\") " Nov 03 20:02:42.401578 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:42.375282 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9423b69f-7ef4-43ed-813e-473355ff28c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9423b69f-7ef4-43ed-813e-473355ff28c7-maas-api-openshift-ca\") pod \"9423b69f-7ef4-43ed-813e-473355ff28c7\" (UID: \"9423b69f-7ef4-43ed-813e-473355ff28c7\") " Nov 03 20:02:42.401578 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:42.400751 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9423b69f-7ef4-43ed-813e-473355ff28c7-maas-api-openshift-ca" pod "9423b69f-7ef4-43ed-813e-473355ff28c7" (UID: "9423b69f-7ef4-43ed-813e-473355ff28c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:42.404095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:42.404044 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9423b69f-7ef4-43ed-813e-473355ff28c7-kube-api-access-zb8qc" pod "9423b69f-7ef4-43ed-813e-473355ff28c7" (UID: "9423b69f-7ef4-43ed-813e-473355ff28c7"). InnerVolumeSpecName "kube-api-access-zb8qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:42.406633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:42.406575 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9423b69f-7ef4-43ed-813e-473355ff28c7-maas-api-xks-ca" pod "9423b69f-7ef4-43ed-813e-473355ff28c7" (UID: "9423b69f-7ef4-43ed-813e-473355ff28c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:42.477557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:42.477205 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9423b69f-7ef4-43ed-813e-473355ff28c7-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:42.477557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:42.477245 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb8qc\" (UniqueName: \"kubernetes.io/projected/9423b69f-7ef4-43ed-813e-473355ff28c7-kube-api-access-zb8qc\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:42.477557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:42.477265 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9423b69f-7ef4-43ed-813e-473355ff28c7-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:42.501555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:42.496915 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:02:42.501555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:42.496993 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" podUID="8bf7d094-ba6e-4c14-b731-c13068585df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:02:42.674239 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:42.673827 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:02:42.674239 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:42.673887 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" podUID="7310a5cf-4706-445f-8c50-95e2f57c1ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 20:02:43.006874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:43.006791 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" Nov 03 20:02:43.006874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:43.006801 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg" event={"ID":"9423b69f-7ef4-43ed-813e-473355ff28c7","Type":"ContainerDied","Data":"05d6f04d1a5d0635d55e1fee1d6d9a2ef1ae9da2290d2ce25394829271210257"} Nov 03 20:02:43.006874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:43.006850 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05d6f04d1a5d0635d55e1fee1d6d9a2ef1ae9da2290d2ce25394829271210257" Nov 03 20:02:43.494923 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:43.494884 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:02:43.495916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:43.495879 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" podUID="8bf7d094-ba6e-4c14-b731-c13068585df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:02:43.679950 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:43.679916 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:02:43.680225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:43.680193 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" podUID="7310a5cf-4706-445f-8c50-95e2f57c1ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 20:02:44.676556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:44.676090 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:02:44.676556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:44.676172 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" podUID="7310a5cf-4706-445f-8c50-95e2f57c1ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 20:02:44.776866 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:44.776382 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:44.777293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:44.777201 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:44.778966 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:44.778943 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:02:45.678468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:45.678437 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:45.679000 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:45.678871 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:45.679558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:45.679511 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:02:49.214207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.214109 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf"] Nov 03 20:02:49.215003 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.214975 5111 cpu_manager.go:414] "containerMap: removing container" podUID="9423b69f-7ef4-43ed-813e-473355ff28c7" containerName="revoke-keys" Nov 03 20:02:49.215143 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.215126 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9423b69f-7ef4-43ed-813e-473355ff28c7" containerName="revoke-keys" Nov 03 20:02:49.215516 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.215492 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="9423b69f-7ef4-43ed-813e-473355ff28c7" containerName="revoke-keys" Nov 03 20:02:49.224597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.221403 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf" Nov 03 20:02:49.236746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.235871 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:02:49.236746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.235936 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:02:49.236746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.236046 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:02:49.236746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.236152 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:02:49.242224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.242177 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf"] Nov 03 20:02:49.269381 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.269339 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7d5n\" (UniqueName: \"kubernetes.io/projected/b3583b36-503b-4421-a11a-8ea6e61bd31f-kube-api-access-b7d5n\") pod \"maas-api-revoke-sub-5cf58a55-wbhcf\" (UID: \"b3583b36-503b-4421-a11a-8ea6e61bd31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf" Nov 03 20:02:49.269574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.269522 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3583b36-503b-4421-a11a-8ea6e61bd31f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf58a55-wbhcf\" (UID: \"b3583b36-503b-4421-a11a-8ea6e61bd31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf" Nov 03 20:02:49.269687 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.269665 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3583b36-503b-4421-a11a-8ea6e61bd31f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf58a55-wbhcf\" (UID: \"b3583b36-503b-4421-a11a-8ea6e61bd31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf" Nov 03 20:02:49.372400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.372367 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3583b36-503b-4421-a11a-8ea6e61bd31f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf58a55-wbhcf\" (UID: \"b3583b36-503b-4421-a11a-8ea6e61bd31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf" Nov 03 20:02:49.372591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.372441 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3583b36-503b-4421-a11a-8ea6e61bd31f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf58a55-wbhcf\" (UID: \"b3583b36-503b-4421-a11a-8ea6e61bd31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf" Nov 03 20:02:49.372591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.372523 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7d5n\" (UniqueName: \"kubernetes.io/projected/b3583b36-503b-4421-a11a-8ea6e61bd31f-kube-api-access-b7d5n\") pod \"maas-api-revoke-sub-5cf58a55-wbhcf\" (UID: \"b3583b36-503b-4421-a11a-8ea6e61bd31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf" Nov 03 20:02:49.373155 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.373132 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3583b36-503b-4421-a11a-8ea6e61bd31f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf58a55-wbhcf\" (UID: \"b3583b36-503b-4421-a11a-8ea6e61bd31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf" Nov 03 20:02:49.374905 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.374879 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3583b36-503b-4421-a11a-8ea6e61bd31f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf58a55-wbhcf\" (UID: \"b3583b36-503b-4421-a11a-8ea6e61bd31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf" Nov 03 20:02:49.380527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.380500 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7d5n\" (UniqueName: \"kubernetes.io/projected/b3583b36-503b-4421-a11a-8ea6e61bd31f-kube-api-access-b7d5n\") pod \"maas-api-revoke-sub-5cf58a55-wbhcf\" (UID: \"b3583b36-503b-4421-a11a-8ea6e61bd31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf" Nov 03 20:02:49.581084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.581011 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf" Nov 03 20:02:49.678559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.677455 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2"] Nov 03 20:02:49.701144 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.701108 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2"] Nov 03 20:02:49.701295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.701237 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" Nov 03 20:02:49.779239 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.779196 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc9e40c-d149-44a0-a0e6-c7081d54e530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3490817-hshq2\" (UID: \"9fc9e40c-d149-44a0-a0e6-c7081d54e530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" Nov 03 20:02:49.779407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.779300 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dmcj\" (UniqueName: \"kubernetes.io/projected/9fc9e40c-d149-44a0-a0e6-c7081d54e530-kube-api-access-6dmcj\") pod \"maas-api-revoke-sub-c3490817-hshq2\" (UID: \"9fc9e40c-d149-44a0-a0e6-c7081d54e530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" Nov 03 20:02:49.779482 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.779462 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc9e40c-d149-44a0-a0e6-c7081d54e530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3490817-hshq2\" (UID: \"9fc9e40c-d149-44a0-a0e6-c7081d54e530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" Nov 03 20:02:49.865360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.865330 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf"] Nov 03 20:02:49.867007 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:02:49.866977 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3583b36_503b_4421_a11a_8ea6e61bd31f.slice/crio-59c6e5a778304c98737a7a336a2e5df8e569c98840c43162c49a418f5011d5d1 WatchSource:0}: Error finding container 59c6e5a778304c98737a7a336a2e5df8e569c98840c43162c49a418f5011d5d1: Status 404 returned error can't find the container with id 59c6e5a778304c98737a7a336a2e5df8e569c98840c43162c49a418f5011d5d1 Nov 03 20:02:49.880454 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.880427 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc9e40c-d149-44a0-a0e6-c7081d54e530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3490817-hshq2\" (UID: \"9fc9e40c-d149-44a0-a0e6-c7081d54e530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" Nov 03 20:02:49.880598 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.880496 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dmcj\" (UniqueName: \"kubernetes.io/projected/9fc9e40c-d149-44a0-a0e6-c7081d54e530-kube-api-access-6dmcj\") pod \"maas-api-revoke-sub-c3490817-hshq2\" (UID: \"9fc9e40c-d149-44a0-a0e6-c7081d54e530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" Nov 03 20:02:49.880673 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.880655 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc9e40c-d149-44a0-a0e6-c7081d54e530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3490817-hshq2\" (UID: \"9fc9e40c-d149-44a0-a0e6-c7081d54e530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" Nov 03 20:02:49.881241 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.881220 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc9e40c-d149-44a0-a0e6-c7081d54e530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3490817-hshq2\" (UID: \"9fc9e40c-d149-44a0-a0e6-c7081d54e530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" Nov 03 20:02:49.882830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.882813 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc9e40c-d149-44a0-a0e6-c7081d54e530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3490817-hshq2\" (UID: \"9fc9e40c-d149-44a0-a0e6-c7081d54e530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" Nov 03 20:02:49.889453 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:49.889432 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dmcj\" (UniqueName: \"kubernetes.io/projected/9fc9e40c-d149-44a0-a0e6-c7081d54e530-kube-api-access-6dmcj\") pod \"maas-api-revoke-sub-c3490817-hshq2\" (UID: \"9fc9e40c-d149-44a0-a0e6-c7081d54e530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" Nov 03 20:02:50.065835 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:50.065802 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" Nov 03 20:02:50.079837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:50.079787 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf" event={"ID":"b3583b36-503b-4421-a11a-8ea6e61bd31f","Type":"ContainerStarted","Data":"59c6e5a778304c98737a7a336a2e5df8e569c98840c43162c49a418f5011d5d1"} Nov 03 20:02:50.367624 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:50.367586 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2"] Nov 03 20:02:50.377496 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:02:50.377472 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc9e40c_d149_44a0_a0e6_c7081d54e530.slice/crio-8fd86cbc8feec9f97d7a884aba603c699fa2824aefb3c647a229b1ed3af2aad6 WatchSource:0}: Error finding container 8fd86cbc8feec9f97d7a884aba603c699fa2824aefb3c647a229b1ed3af2aad6: Status 404 returned error can't find the container with id 8fd86cbc8feec9f97d7a884aba603c699fa2824aefb3c647a229b1ed3af2aad6 Nov 03 20:02:51.087732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:51.087635 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" event={"ID":"9fc9e40c-d149-44a0-a0e6-c7081d54e530","Type":"ContainerStarted","Data":"239bc6379a457163e4a618c4a5c5f881ef622a8efc04f9e9e0b0ec8dd0a75a93"} Nov 03 20:02:51.087732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:51.087706 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" event={"ID":"9fc9e40c-d149-44a0-a0e6-c7081d54e530","Type":"ContainerStarted","Data":"8fd86cbc8feec9f97d7a884aba603c699fa2824aefb3c647a229b1ed3af2aad6"} Nov 03 20:02:51.089176 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:51.089147 5111 generic.go:356] "Generic (PLEG): container finished" podID="b3583b36-503b-4421-a11a-8ea6e61bd31f" containerID="d22be5826698be97336a7022df2962d34d3f7150ef64d8935895d1903c0f1b6b" exitCode=0 Nov 03 20:02:51.089300 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:51.089195 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf" event={"ID":"b3583b36-503b-4421-a11a-8ea6e61bd31f","Type":"ContainerDied","Data":"d22be5826698be97336a7022df2962d34d3f7150ef64d8935895d1903c0f1b6b"} Nov 03 20:02:51.104299 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:51.104255 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" podStartSLOduration=2.104244254 podStartE2EDuration="2.104244254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:51.102302797 +0000 UTC m=+1401.064681234" watchObservedRunningTime="2026-11-03 20:02:51.104244254 +0000 UTC m=+1401.066622691" Nov 03 20:02:51.898089 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:51.898045 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv"] Nov 03 20:02:51.905897 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:51.905865 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" Nov 03 20:02:51.910990 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:51.910959 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv"] Nov 03 20:02:51.928447 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:51.928412 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fa764a6-tlgbv\" (UID: \"cabe4bba-89d1-46b0-8ced-a8cdb34383e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" Nov 03 20:02:51.928654 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:51.928495 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fa764a6-tlgbv\" (UID: \"cabe4bba-89d1-46b0-8ced-a8cdb34383e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" Nov 03 20:02:51.928654 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:51.928644 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9vgg\" (UniqueName: \"kubernetes.io/projected/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-kube-api-access-c9vgg\") pod \"maas-api-revoke-sub-5fa764a6-tlgbv\" (UID: \"cabe4bba-89d1-46b0-8ced-a8cdb34383e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" Nov 03 20:02:52.029372 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.029336 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9vgg\" (UniqueName: \"kubernetes.io/projected/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-kube-api-access-c9vgg\") pod \"maas-api-revoke-sub-5fa764a6-tlgbv\" (UID: \"cabe4bba-89d1-46b0-8ced-a8cdb34383e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" Nov 03 20:02:52.029549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.029436 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fa764a6-tlgbv\" (UID: \"cabe4bba-89d1-46b0-8ced-a8cdb34383e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" Nov 03 20:02:52.029549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.029469 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fa764a6-tlgbv\" (UID: \"cabe4bba-89d1-46b0-8ced-a8cdb34383e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" Nov 03 20:02:52.030136 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.030118 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fa764a6-tlgbv\" (UID: \"cabe4bba-89d1-46b0-8ced-a8cdb34383e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" Nov 03 20:02:52.031780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.031755 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fa764a6-tlgbv\" (UID: \"cabe4bba-89d1-46b0-8ced-a8cdb34383e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" Nov 03 20:02:52.038390 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.038362 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9vgg\" (UniqueName: \"kubernetes.io/projected/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-kube-api-access-c9vgg\") pod \"maas-api-revoke-sub-5fa764a6-tlgbv\" (UID: \"cabe4bba-89d1-46b0-8ced-a8cdb34383e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" Nov 03 20:02:52.104267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.099724 5111 generic.go:356] "Generic (PLEG): container finished" podID="9fc9e40c-d149-44a0-a0e6-c7081d54e530" containerID="239bc6379a457163e4a618c4a5c5f881ef622a8efc04f9e9e0b0ec8dd0a75a93" exitCode=0 Nov 03 20:02:52.104267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.100368 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" event={"ID":"9fc9e40c-d149-44a0-a0e6-c7081d54e530","Type":"ContainerDied","Data":"239bc6379a457163e4a618c4a5c5f881ef622a8efc04f9e9e0b0ec8dd0a75a93"} Nov 03 20:02:52.237041 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.237001 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" Nov 03 20:02:52.300162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.300128 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf" Nov 03 20:02:52.351344 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.350155 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3583b36-503b-4421-a11a-8ea6e61bd31f-kube-api-access-b7d5n\" (UniqueName: \"kubernetes.io/projected/b3583b36-503b-4421-a11a-8ea6e61bd31f-kube-api-access-b7d5n\") pod \"b3583b36-503b-4421-a11a-8ea6e61bd31f\" (UID: \"b3583b36-503b-4421-a11a-8ea6e61bd31f\") " Nov 03 20:02:52.351344 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.350233 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3583b36-503b-4421-a11a-8ea6e61bd31f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3583b36-503b-4421-a11a-8ea6e61bd31f-maas-api-openshift-ca\") pod \"b3583b36-503b-4421-a11a-8ea6e61bd31f\" (UID: \"b3583b36-503b-4421-a11a-8ea6e61bd31f\") " Nov 03 20:02:52.351344 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.350314 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3583b36-503b-4421-a11a-8ea6e61bd31f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3583b36-503b-4421-a11a-8ea6e61bd31f-maas-api-xks-ca\") pod \"b3583b36-503b-4421-a11a-8ea6e61bd31f\" (UID: \"b3583b36-503b-4421-a11a-8ea6e61bd31f\") " Nov 03 20:02:52.375550 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.369250 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3583b36-503b-4421-a11a-8ea6e61bd31f-maas-api-openshift-ca" pod "b3583b36-503b-4421-a11a-8ea6e61bd31f" (UID: "b3583b36-503b-4421-a11a-8ea6e61bd31f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:52.375550 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.372705 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3583b36-503b-4421-a11a-8ea6e61bd31f-maas-api-xks-ca" pod "b3583b36-503b-4421-a11a-8ea6e61bd31f" (UID: "b3583b36-503b-4421-a11a-8ea6e61bd31f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:52.395082 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.395032 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3583b36-503b-4421-a11a-8ea6e61bd31f-kube-api-access-b7d5n" pod "b3583b36-503b-4421-a11a-8ea6e61bd31f" (UID: "b3583b36-503b-4421-a11a-8ea6e61bd31f"). InnerVolumeSpecName "kube-api-access-b7d5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:52.474201 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.474167 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3583b36-503b-4421-a11a-8ea6e61bd31f-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:52.474338 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.474206 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7d5n\" (UniqueName: \"kubernetes.io/projected/b3583b36-503b-4421-a11a-8ea6e61bd31f-kube-api-access-b7d5n\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:52.474338 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.474224 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3583b36-503b-4421-a11a-8ea6e61bd31f-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:52.559569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:52.556766 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv"] Nov 03 20:02:52.559569 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:02:52.557607 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabe4bba_89d1_46b0_8ced_a8cdb34383e2.slice/crio-6d2d884d2f103de2d3c4708f1ddd1f330e8ceac22f92ad249ac0fdb08a2dd0bc WatchSource:0}: Error finding container 6d2d884d2f103de2d3c4708f1ddd1f330e8ceac22f92ad249ac0fdb08a2dd0bc: Status 404 returned error can't find the container with id 6d2d884d2f103de2d3c4708f1ddd1f330e8ceac22f92ad249ac0fdb08a2dd0bc Nov 03 20:02:53.112394 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:53.112311 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf" Nov 03 20:02:53.112394 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:53.112334 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf" event={"ID":"b3583b36-503b-4421-a11a-8ea6e61bd31f","Type":"ContainerDied","Data":"59c6e5a778304c98737a7a336a2e5df8e569c98840c43162c49a418f5011d5d1"} Nov 03 20:02:53.112933 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:53.112394 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59c6e5a778304c98737a7a336a2e5df8e569c98840c43162c49a418f5011d5d1" Nov 03 20:02:53.114395 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:53.114354 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" event={"ID":"cabe4bba-89d1-46b0-8ced-a8cdb34383e2","Type":"ContainerStarted","Data":"5c162edccce3381f64f00187b93347949b59e847ed155d54b16eabc79c03a626"} Nov 03 20:02:53.114508 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:53.114410 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" event={"ID":"cabe4bba-89d1-46b0-8ced-a8cdb34383e2","Type":"ContainerStarted","Data":"6d2d884d2f103de2d3c4708f1ddd1f330e8ceac22f92ad249ac0fdb08a2dd0bc"} Nov 03 20:02:53.133385 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:53.133341 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" podStartSLOduration=2.133327757 podStartE2EDuration="2.133327757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:53.131085061 +0000 UTC m=+1403.093463501" watchObservedRunningTime="2026-11-03 20:02:53.133327757 +0000 UTC m=+1403.095706194" Nov 03 20:02:53.295549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:53.295508 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" Nov 03 20:02:53.400556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:53.395333 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fc9e40c-d149-44a0-a0e6-c7081d54e530-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc9e40c-d149-44a0-a0e6-c7081d54e530-maas-api-xks-ca\") pod \"9fc9e40c-d149-44a0-a0e6-c7081d54e530\" (UID: \"9fc9e40c-d149-44a0-a0e6-c7081d54e530\") " Nov 03 20:02:53.400556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:53.395436 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc9e40c-d149-44a0-a0e6-c7081d54e530-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc9e40c-d149-44a0-a0e6-c7081d54e530-maas-api-openshift-ca\") pod \"9fc9e40c-d149-44a0-a0e6-c7081d54e530\" (UID: \"9fc9e40c-d149-44a0-a0e6-c7081d54e530\") " Nov 03 20:02:53.400556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:53.395558 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc9e40c-d149-44a0-a0e6-c7081d54e530-kube-api-access-6dmcj\" (UniqueName: \"kubernetes.io/projected/9fc9e40c-d149-44a0-a0e6-c7081d54e530-kube-api-access-6dmcj\") pod \"9fc9e40c-d149-44a0-a0e6-c7081d54e530\" (UID: \"9fc9e40c-d149-44a0-a0e6-c7081d54e530\") " Nov 03 20:02:53.400556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:53.397256 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc9e40c-d149-44a0-a0e6-c7081d54e530-maas-api-openshift-ca" pod "9fc9e40c-d149-44a0-a0e6-c7081d54e530" (UID: "9fc9e40c-d149-44a0-a0e6-c7081d54e530"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:53.419761 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:53.419727 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc9e40c-d149-44a0-a0e6-c7081d54e530-kube-api-access-6dmcj" pod "9fc9e40c-d149-44a0-a0e6-c7081d54e530" (UID: "9fc9e40c-d149-44a0-a0e6-c7081d54e530"). InnerVolumeSpecName "kube-api-access-6dmcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:53.420798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:53.420181 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc9e40c-d149-44a0-a0e6-c7081d54e530-maas-api-xks-ca" pod "9fc9e40c-d149-44a0-a0e6-c7081d54e530" (UID: "9fc9e40c-d149-44a0-a0e6-c7081d54e530"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:53.500577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:53.500541 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc9e40c-d149-44a0-a0e6-c7081d54e530-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:53.500577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:53.500577 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc9e40c-d149-44a0-a0e6-c7081d54e530-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:53.500791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:53.500592 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dmcj\" (UniqueName: \"kubernetes.io/projected/9fc9e40c-d149-44a0-a0e6-c7081d54e530-kube-api-access-6dmcj\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:54.122767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.122734 5111 generic.go:356] "Generic (PLEG): container finished" podID="cabe4bba-89d1-46b0-8ced-a8cdb34383e2" containerID="5c162edccce3381f64f00187b93347949b59e847ed155d54b16eabc79c03a626" exitCode=0 Nov 03 20:02:54.123199 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.122817 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" event={"ID":"cabe4bba-89d1-46b0-8ced-a8cdb34383e2","Type":"ContainerDied","Data":"5c162edccce3381f64f00187b93347949b59e847ed155d54b16eabc79c03a626"} Nov 03 20:02:54.124341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.124313 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" Nov 03 20:02:54.124341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.124331 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2" event={"ID":"9fc9e40c-d149-44a0-a0e6-c7081d54e530","Type":"ContainerDied","Data":"8fd86cbc8feec9f97d7a884aba603c699fa2824aefb3c647a229b1ed3af2aad6"} Nov 03 20:02:54.124489 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.124370 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fd86cbc8feec9f97d7a884aba603c699fa2824aefb3c647a229b1ed3af2aad6" Nov 03 20:02:54.297649 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.297615 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh"] Nov 03 20:02:54.298184 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.298171 5111 cpu_manager.go:414] "containerMap: removing container" podUID="9fc9e40c-d149-44a0-a0e6-c7081d54e530" containerName="revoke-keys" Nov 03 20:02:54.298231 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.298185 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc9e40c-d149-44a0-a0e6-c7081d54e530" containerName="revoke-keys" Nov 03 20:02:54.298231 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.298224 5111 cpu_manager.go:414] "containerMap: removing container" podUID="b3583b36-503b-4421-a11a-8ea6e61bd31f" containerName="revoke-keys" Nov 03 20:02:54.298231 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.298230 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3583b36-503b-4421-a11a-8ea6e61bd31f" containerName="revoke-keys" Nov 03 20:02:54.298342 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.298332 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc9e40c-d149-44a0-a0e6-c7081d54e530" containerName="revoke-keys" Nov 03 20:02:54.298378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.298348 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3583b36-503b-4421-a11a-8ea6e61bd31f" containerName="revoke-keys" Nov 03 20:02:54.301820 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.301805 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh" Nov 03 20:02:54.307946 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.307921 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh"] Nov 03 20:02:54.414899 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.414820 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6427e57d-fa48-49c8-a6a4-dca0994f967a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f95ec60-sp9kh\" (UID: \"6427e57d-fa48-49c8-a6a4-dca0994f967a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh" Nov 03 20:02:54.414899 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.414870 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6427e57d-fa48-49c8-a6a4-dca0994f967a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f95ec60-sp9kh\" (UID: \"6427e57d-fa48-49c8-a6a4-dca0994f967a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh" Nov 03 20:02:54.415080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.415048 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhqrg\" (UniqueName: \"kubernetes.io/projected/6427e57d-fa48-49c8-a6a4-dca0994f967a-kube-api-access-nhqrg\") pod \"maas-api-revoke-sub-6f95ec60-sp9kh\" (UID: \"6427e57d-fa48-49c8-a6a4-dca0994f967a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh" Nov 03 20:02:54.517555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.517159 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhqrg\" (UniqueName: \"kubernetes.io/projected/6427e57d-fa48-49c8-a6a4-dca0994f967a-kube-api-access-nhqrg\") pod \"maas-api-revoke-sub-6f95ec60-sp9kh\" (UID: \"6427e57d-fa48-49c8-a6a4-dca0994f967a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh" Nov 03 20:02:54.517555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.517308 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6427e57d-fa48-49c8-a6a4-dca0994f967a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f95ec60-sp9kh\" (UID: \"6427e57d-fa48-49c8-a6a4-dca0994f967a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh" Nov 03 20:02:54.517555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.517369 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6427e57d-fa48-49c8-a6a4-dca0994f967a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f95ec60-sp9kh\" (UID: \"6427e57d-fa48-49c8-a6a4-dca0994f967a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh" Nov 03 20:02:54.520416 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.518346 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6427e57d-fa48-49c8-a6a4-dca0994f967a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f95ec60-sp9kh\" (UID: \"6427e57d-fa48-49c8-a6a4-dca0994f967a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh" Nov 03 20:02:54.520941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.520914 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6427e57d-fa48-49c8-a6a4-dca0994f967a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f95ec60-sp9kh\" (UID: \"6427e57d-fa48-49c8-a6a4-dca0994f967a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh" Nov 03 20:02:54.525213 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.525187 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhqrg\" (UniqueName: \"kubernetes.io/projected/6427e57d-fa48-49c8-a6a4-dca0994f967a-kube-api-access-nhqrg\") pod \"maas-api-revoke-sub-6f95ec60-sp9kh\" (UID: \"6427e57d-fa48-49c8-a6a4-dca0994f967a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh" Nov 03 20:02:54.617381 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.617343 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh" Nov 03 20:02:54.903400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:54.903366 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh"] Nov 03 20:02:54.904395 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:02:54.904366 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6427e57d_fa48_49c8_a6a4_dca0994f967a.slice/crio-9b6812b6faf1269331a4e61abd55f97b96e56aacacd58bc60765e277e2437074 WatchSource:0}: Error finding container 9b6812b6faf1269331a4e61abd55f97b96e56aacacd58bc60765e277e2437074: Status 404 returned error can't find the container with id 9b6812b6faf1269331a4e61abd55f97b96e56aacacd58bc60765e277e2437074 Nov 03 20:02:55.132975 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:55.132934 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh" event={"ID":"6427e57d-fa48-49c8-a6a4-dca0994f967a","Type":"ContainerStarted","Data":"9b6812b6faf1269331a4e61abd55f97b96e56aacacd58bc60765e277e2437074"} Nov 03 20:02:55.294528 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:55.294506 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" Nov 03 20:02:55.429089 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:55.429049 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-kube-api-access-c9vgg\" (UniqueName: \"kubernetes.io/projected/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-kube-api-access-c9vgg\") pod \"cabe4bba-89d1-46b0-8ced-a8cdb34383e2\" (UID: \"cabe4bba-89d1-46b0-8ced-a8cdb34383e2\") " Nov 03 20:02:55.429255 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:55.429094 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-maas-api-openshift-ca\") pod \"cabe4bba-89d1-46b0-8ced-a8cdb34383e2\" (UID: \"cabe4bba-89d1-46b0-8ced-a8cdb34383e2\") " Nov 03 20:02:55.429255 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:55.429246 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-maas-api-xks-ca\") pod \"cabe4bba-89d1-46b0-8ced-a8cdb34383e2\" (UID: \"cabe4bba-89d1-46b0-8ced-a8cdb34383e2\") " Nov 03 20:02:55.429488 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:55.429463 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-maas-api-openshift-ca" pod "cabe4bba-89d1-46b0-8ced-a8cdb34383e2" (UID: "cabe4bba-89d1-46b0-8ced-a8cdb34383e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:55.429675 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:55.429660 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:55.431318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:55.431293 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-maas-api-xks-ca" pod "cabe4bba-89d1-46b0-8ced-a8cdb34383e2" (UID: "cabe4bba-89d1-46b0-8ced-a8cdb34383e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:55.431443 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:55.431345 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-kube-api-access-c9vgg" pod "cabe4bba-89d1-46b0-8ced-a8cdb34383e2" (UID: "cabe4bba-89d1-46b0-8ced-a8cdb34383e2"). InnerVolumeSpecName "kube-api-access-c9vgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:55.535019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:55.530756 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9vgg\" (UniqueName: \"kubernetes.io/projected/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-kube-api-access-c9vgg\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:55.535019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:55.530794 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabe4bba-89d1-46b0-8ced-a8cdb34383e2-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:56.141203 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:56.141166 5111 generic.go:356] "Generic (PLEG): container finished" podID="6427e57d-fa48-49c8-a6a4-dca0994f967a" containerID="7ed457a3a1c41c29543fd8ec45d98fa48d60761c13539ffc477fc4444dc91adb" exitCode=0 Nov 03 20:02:56.141688 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:56.141251 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh" event={"ID":"6427e57d-fa48-49c8-a6a4-dca0994f967a","Type":"ContainerDied","Data":"7ed457a3a1c41c29543fd8ec45d98fa48d60761c13539ffc477fc4444dc91adb"} Nov 03 20:02:56.142788 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:56.142769 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" Nov 03 20:02:56.142895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:56.142770 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv" event={"ID":"cabe4bba-89d1-46b0-8ced-a8cdb34383e2","Type":"ContainerDied","Data":"6d2d884d2f103de2d3c4708f1ddd1f330e8ceac22f92ad249ac0fdb08a2dd0bc"} Nov 03 20:02:56.142937 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:56.142893 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d2d884d2f103de2d3c4708f1ddd1f330e8ceac22f92ad249ac0fdb08a2dd0bc" Nov 03 20:02:57.408423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:57.408348 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh" Nov 03 20:02:57.571797 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:57.571764 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6427e57d-fa48-49c8-a6a4-dca0994f967a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6427e57d-fa48-49c8-a6a4-dca0994f967a-maas-api-openshift-ca\") pod \"6427e57d-fa48-49c8-a6a4-dca0994f967a\" (UID: \"6427e57d-fa48-49c8-a6a4-dca0994f967a\") " Nov 03 20:02:57.572018 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:57.571935 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6427e57d-fa48-49c8-a6a4-dca0994f967a-kube-api-access-nhqrg\" (UniqueName: \"kubernetes.io/projected/6427e57d-fa48-49c8-a6a4-dca0994f967a-kube-api-access-nhqrg\") pod \"6427e57d-fa48-49c8-a6a4-dca0994f967a\" (UID: \"6427e57d-fa48-49c8-a6a4-dca0994f967a\") " Nov 03 20:02:57.572086 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:57.571996 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6427e57d-fa48-49c8-a6a4-dca0994f967a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6427e57d-fa48-49c8-a6a4-dca0994f967a-maas-api-xks-ca\") pod \"6427e57d-fa48-49c8-a6a4-dca0994f967a\" (UID: \"6427e57d-fa48-49c8-a6a4-dca0994f967a\") " Nov 03 20:02:57.572149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:57.572114 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6427e57d-fa48-49c8-a6a4-dca0994f967a-maas-api-openshift-ca" pod "6427e57d-fa48-49c8-a6a4-dca0994f967a" (UID: "6427e57d-fa48-49c8-a6a4-dca0994f967a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:57.572518 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:57.572494 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6427e57d-fa48-49c8-a6a4-dca0994f967a-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:57.574468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:57.574429 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6427e57d-fa48-49c8-a6a4-dca0994f967a-maas-api-xks-ca" pod "6427e57d-fa48-49c8-a6a4-dca0994f967a" (UID: "6427e57d-fa48-49c8-a6a4-dca0994f967a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:57.583004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:57.582980 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6427e57d-fa48-49c8-a6a4-dca0994f967a-kube-api-access-nhqrg" pod "6427e57d-fa48-49c8-a6a4-dca0994f967a" (UID: "6427e57d-fa48-49c8-a6a4-dca0994f967a"). InnerVolumeSpecName "kube-api-access-nhqrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:57.694622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:57.679470 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhqrg\" (UniqueName: \"kubernetes.io/projected/6427e57d-fa48-49c8-a6a4-dca0994f967a-kube-api-access-nhqrg\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:57.694622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:57.679506 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6427e57d-fa48-49c8-a6a4-dca0994f967a-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:58.227789 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.226925 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh" event={"ID":"6427e57d-fa48-49c8-a6a4-dca0994f967a","Type":"ContainerDied","Data":"9b6812b6faf1269331a4e61abd55f97b96e56aacacd58bc60765e277e2437074"} Nov 03 20:02:58.227789 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.226992 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b6812b6faf1269331a4e61abd55f97b96e56aacacd58bc60765e277e2437074" Nov 03 20:02:58.227789 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.227066 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh" Nov 03 20:02:58.806173 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.806139 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c"] Nov 03 20:02:58.806766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.806749 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6427e57d-fa48-49c8-a6a4-dca0994f967a" containerName="revoke-keys" Nov 03 20:02:58.806766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.806766 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6427e57d-fa48-49c8-a6a4-dca0994f967a" containerName="revoke-keys" Nov 03 20:02:58.806854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.806782 5111 cpu_manager.go:414] "containerMap: removing container" podUID="cabe4bba-89d1-46b0-8ced-a8cdb34383e2" containerName="revoke-keys" Nov 03 20:02:58.806854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.806788 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabe4bba-89d1-46b0-8ced-a8cdb34383e2" containerName="revoke-keys" Nov 03 20:02:58.806919 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.806887 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabe4bba-89d1-46b0-8ced-a8cdb34383e2" containerName="revoke-keys" Nov 03 20:02:58.806919 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.806897 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6427e57d-fa48-49c8-a6a4-dca0994f967a" containerName="revoke-keys" Nov 03 20:02:58.810527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.810511 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" Nov 03 20:02:58.815401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.815375 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:02:58.815401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.815387 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:02:58.815564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.815431 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:02:58.815564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.815469 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:02:58.817818 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.817656 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c"] Nov 03 20:02:58.914553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.914494 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjlg5\" (UniqueName: \"kubernetes.io/projected/734a8dba-3f4c-4c43-adea-556a98c0d057-kube-api-access-cjlg5\") pod \"maas-api-revoke-sub-f65635a0-wtq9c\" (UID: \"734a8dba-3f4c-4c43-adea-556a98c0d057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" Nov 03 20:02:58.914751 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.914676 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734a8dba-3f4c-4c43-adea-556a98c0d057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f65635a0-wtq9c\" (UID: \"734a8dba-3f4c-4c43-adea-556a98c0d057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" Nov 03 20:02:58.914849 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:58.914826 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734a8dba-3f4c-4c43-adea-556a98c0d057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f65635a0-wtq9c\" (UID: \"734a8dba-3f4c-4c43-adea-556a98c0d057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" Nov 03 20:02:59.016025 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:59.015992 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734a8dba-3f4c-4c43-adea-556a98c0d057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f65635a0-wtq9c\" (UID: \"734a8dba-3f4c-4c43-adea-556a98c0d057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" Nov 03 20:02:59.016191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:59.016084 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjlg5\" (UniqueName: \"kubernetes.io/projected/734a8dba-3f4c-4c43-adea-556a98c0d057-kube-api-access-cjlg5\") pod \"maas-api-revoke-sub-f65635a0-wtq9c\" (UID: \"734a8dba-3f4c-4c43-adea-556a98c0d057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" Nov 03 20:02:59.016191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:59.016122 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734a8dba-3f4c-4c43-adea-556a98c0d057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f65635a0-wtq9c\" (UID: \"734a8dba-3f4c-4c43-adea-556a98c0d057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" Nov 03 20:02:59.016760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:59.016740 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734a8dba-3f4c-4c43-adea-556a98c0d057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f65635a0-wtq9c\" (UID: \"734a8dba-3f4c-4c43-adea-556a98c0d057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" Nov 03 20:02:59.018378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:59.018356 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734a8dba-3f4c-4c43-adea-556a98c0d057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f65635a0-wtq9c\" (UID: \"734a8dba-3f4c-4c43-adea-556a98c0d057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" Nov 03 20:02:59.023772 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:59.023750 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjlg5\" (UniqueName: \"kubernetes.io/projected/734a8dba-3f4c-4c43-adea-556a98c0d057-kube-api-access-cjlg5\") pod \"maas-api-revoke-sub-f65635a0-wtq9c\" (UID: \"734a8dba-3f4c-4c43-adea-556a98c0d057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" Nov 03 20:02:59.127077 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:59.127048 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" Nov 03 20:02:59.406484 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:02:59.406450 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c"] Nov 03 20:02:59.408148 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:02:59.408124 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734a8dba_3f4c_4c43_adea_556a98c0d057.slice/crio-d2f8e835b670433b7f0abc7e236ba68165ccbe02a0d89b2e19541179ed6597a7 WatchSource:0}: Error finding container d2f8e835b670433b7f0abc7e236ba68165ccbe02a0d89b2e19541179ed6597a7: Status 404 returned error can't find the container with id d2f8e835b670433b7f0abc7e236ba68165ccbe02a0d89b2e19541179ed6597a7 Nov 03 20:03:00.254605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:00.254563 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" event={"ID":"734a8dba-3f4c-4c43-adea-556a98c0d057","Type":"ContainerStarted","Data":"463b5e920fd3d14306f88d47dbd587a50e5f34f8c538fb01415d5c1c8fb9a8ad"} Nov 03 20:03:00.255031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:00.254643 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" event={"ID":"734a8dba-3f4c-4c43-adea-556a98c0d057","Type":"ContainerStarted","Data":"d2f8e835b670433b7f0abc7e236ba68165ccbe02a0d89b2e19541179ed6597a7"} Nov 03 20:03:00.273022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:00.272976 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" podStartSLOduration=2.272963071 podStartE2EDuration="2.272963071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:00.270001123 +0000 UTC m=+1410.232379560" watchObservedRunningTime="2026-11-03 20:03:00.272963071 +0000 UTC m=+1410.235341508" Nov 03 20:03:01.262992 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:01.262954 5111 generic.go:356] "Generic (PLEG): container finished" podID="734a8dba-3f4c-4c43-adea-556a98c0d057" containerID="463b5e920fd3d14306f88d47dbd587a50e5f34f8c538fb01415d5c1c8fb9a8ad" exitCode=0 Nov 03 20:03:01.263490 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:01.263041 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" event={"ID":"734a8dba-3f4c-4c43-adea-556a98c0d057","Type":"ContainerDied","Data":"463b5e920fd3d14306f88d47dbd587a50e5f34f8c538fb01415d5c1c8fb9a8ad"} Nov 03 20:03:02.404720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.404692 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t"] Nov 03 20:03:02.409396 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.409371 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t" Nov 03 20:03:02.415236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.415200 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t"] Nov 03 20:03:02.459805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.459728 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d01765e-vds4t\" (UID: \"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t" Nov 03 20:03:02.460469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.460257 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d01765e-vds4t\" (UID: \"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t" Nov 03 20:03:02.460469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.460324 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbf2m\" (UniqueName: \"kubernetes.io/projected/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-kube-api-access-kbf2m\") pod \"maas-api-revoke-sub-1d01765e-vds4t\" (UID: \"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t" Nov 03 20:03:02.499565 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.497768 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" Nov 03 20:03:02.561651 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.561558 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/734a8dba-3f4c-4c43-adea-556a98c0d057-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734a8dba-3f4c-4c43-adea-556a98c0d057-maas-api-openshift-ca\") pod \"734a8dba-3f4c-4c43-adea-556a98c0d057\" (UID: \"734a8dba-3f4c-4c43-adea-556a98c0d057\") " Nov 03 20:03:02.561809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.561784 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734a8dba-3f4c-4c43-adea-556a98c0d057-kube-api-access-cjlg5\" (UniqueName: \"kubernetes.io/projected/734a8dba-3f4c-4c43-adea-556a98c0d057-kube-api-access-cjlg5\") pod \"734a8dba-3f4c-4c43-adea-556a98c0d057\" (UID: \"734a8dba-3f4c-4c43-adea-556a98c0d057\") " Nov 03 20:03:02.561913 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.561899 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/734a8dba-3f4c-4c43-adea-556a98c0d057-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734a8dba-3f4c-4c43-adea-556a98c0d057-maas-api-xks-ca\") pod \"734a8dba-3f4c-4c43-adea-556a98c0d057\" (UID: \"734a8dba-3f4c-4c43-adea-556a98c0d057\") " Nov 03 20:03:02.561983 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.561956 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734a8dba-3f4c-4c43-adea-556a98c0d057-maas-api-openshift-ca" pod "734a8dba-3f4c-4c43-adea-556a98c0d057" (UID: "734a8dba-3f4c-4c43-adea-556a98c0d057"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:02.562085 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.562064 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbf2m\" (UniqueName: \"kubernetes.io/projected/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-kube-api-access-kbf2m\") pod \"maas-api-revoke-sub-1d01765e-vds4t\" (UID: \"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t" Nov 03 20:03:02.562196 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.562180 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d01765e-vds4t\" (UID: \"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t" Nov 03 20:03:02.562395 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.562367 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d01765e-vds4t\" (UID: \"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t" Nov 03 20:03:02.562569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.562444 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734a8dba-3f4c-4c43-adea-556a98c0d057-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:02.563471 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.563442 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d01765e-vds4t\" (UID: \"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t" Nov 03 20:03:02.564344 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.564322 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734a8dba-3f4c-4c43-adea-556a98c0d057-kube-api-access-cjlg5" pod "734a8dba-3f4c-4c43-adea-556a98c0d057" (UID: "734a8dba-3f4c-4c43-adea-556a98c0d057"). InnerVolumeSpecName "kube-api-access-cjlg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:02.564563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.564509 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734a8dba-3f4c-4c43-adea-556a98c0d057-maas-api-xks-ca" pod "734a8dba-3f4c-4c43-adea-556a98c0d057" (UID: "734a8dba-3f4c-4c43-adea-556a98c0d057"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:02.565136 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.565119 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d01765e-vds4t\" (UID: \"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t" Nov 03 20:03:02.570096 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.570075 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbf2m\" (UniqueName: \"kubernetes.io/projected/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-kube-api-access-kbf2m\") pod \"maas-api-revoke-sub-1d01765e-vds4t\" (UID: \"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t" Nov 03 20:03:02.663999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.663958 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjlg5\" (UniqueName: \"kubernetes.io/projected/734a8dba-3f4c-4c43-adea-556a98c0d057-kube-api-access-cjlg5\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:02.663999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.663995 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734a8dba-3f4c-4c43-adea-556a98c0d057-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:02.683196 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.683160 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn"] Nov 03 20:03:02.684108 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.684091 5111 cpu_manager.go:414] "containerMap: removing container" podUID="734a8dba-3f4c-4c43-adea-556a98c0d057" containerName="revoke-keys" Nov 03 20:03:02.684161 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.684112 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734a8dba-3f4c-4c43-adea-556a98c0d057" containerName="revoke-keys" Nov 03 20:03:02.684277 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.684266 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="734a8dba-3f4c-4c43-adea-556a98c0d057" containerName="revoke-keys" Nov 03 20:03:02.688847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.688826 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn" Nov 03 20:03:02.694339 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.694205 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn"] Nov 03 20:03:02.766414 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.766377 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efe8399-943b-48ac-8555-72198abc6c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4116667c-vtvbn\" (UID: \"7efe8399-943b-48ac-8555-72198abc6c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn" Nov 03 20:03:02.766707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.766686 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pqcn\" (UniqueName: \"kubernetes.io/projected/7efe8399-943b-48ac-8555-72198abc6c9d-kube-api-access-5pqcn\") pod \"maas-api-revoke-sub-4116667c-vtvbn\" (UID: \"7efe8399-943b-48ac-8555-72198abc6c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn" Nov 03 20:03:02.766842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.766754 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efe8399-943b-48ac-8555-72198abc6c9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4116667c-vtvbn\" (UID: \"7efe8399-943b-48ac-8555-72198abc6c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn" Nov 03 20:03:02.797060 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.797014 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t" Nov 03 20:03:02.868568 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.868508 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pqcn\" (UniqueName: \"kubernetes.io/projected/7efe8399-943b-48ac-8555-72198abc6c9d-kube-api-access-5pqcn\") pod \"maas-api-revoke-sub-4116667c-vtvbn\" (UID: \"7efe8399-943b-48ac-8555-72198abc6c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn" Nov 03 20:03:02.868719 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.868623 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efe8399-943b-48ac-8555-72198abc6c9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4116667c-vtvbn\" (UID: \"7efe8399-943b-48ac-8555-72198abc6c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn" Nov 03 20:03:02.868783 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.868714 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efe8399-943b-48ac-8555-72198abc6c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4116667c-vtvbn\" (UID: \"7efe8399-943b-48ac-8555-72198abc6c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn" Nov 03 20:03:02.869638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.869605 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efe8399-943b-48ac-8555-72198abc6c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4116667c-vtvbn\" (UID: \"7efe8399-943b-48ac-8555-72198abc6c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn" Nov 03 20:03:02.875559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.875518 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efe8399-943b-48ac-8555-72198abc6c9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4116667c-vtvbn\" (UID: \"7efe8399-943b-48ac-8555-72198abc6c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn" Nov 03 20:03:02.878824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.878795 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pqcn\" (UniqueName: \"kubernetes.io/projected/7efe8399-943b-48ac-8555-72198abc6c9d-kube-api-access-5pqcn\") pod \"maas-api-revoke-sub-4116667c-vtvbn\" (UID: \"7efe8399-943b-48ac-8555-72198abc6c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn" Nov 03 20:03:02.997305 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:02.997266 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg"] Nov 03 20:03:03.002802 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.002783 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" Nov 03 20:03:03.007299 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.007270 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn" Nov 03 20:03:03.010621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.009871 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg"] Nov 03 20:03:03.071365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.071305 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvvhb\" (UniqueName: \"kubernetes.io/projected/680aaedd-5d27-4058-a444-b11825314d65-kube-api-access-zvvhb\") pod \"maas-api-revoke-sub-7340dff2-hpcmg\" (UID: \"680aaedd-5d27-4058-a444-b11825314d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" Nov 03 20:03:03.071525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.071427 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680aaedd-5d27-4058-a444-b11825314d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7340dff2-hpcmg\" (UID: \"680aaedd-5d27-4058-a444-b11825314d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" Nov 03 20:03:03.071693 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.071663 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680aaedd-5d27-4058-a444-b11825314d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7340dff2-hpcmg\" (UID: \"680aaedd-5d27-4058-a444-b11825314d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" Nov 03 20:03:03.080301 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:03:03.080271 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c82d407_70a8_4f02_a62e_c3c6f0cbd3da.slice/crio-85c9313763f860df311cea4f42011bb071057f51073413de4f22186fc050e546 WatchSource:0}: Error finding container 85c9313763f860df311cea4f42011bb071057f51073413de4f22186fc050e546: Status 404 returned error can't find the container with id 85c9313763f860df311cea4f42011bb071057f51073413de4f22186fc050e546 Nov 03 20:03:03.080916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.080882 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t"] Nov 03 20:03:03.173349 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.173310 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvvhb\" (UniqueName: \"kubernetes.io/projected/680aaedd-5d27-4058-a444-b11825314d65-kube-api-access-zvvhb\") pod \"maas-api-revoke-sub-7340dff2-hpcmg\" (UID: \"680aaedd-5d27-4058-a444-b11825314d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" Nov 03 20:03:03.173543 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.173403 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680aaedd-5d27-4058-a444-b11825314d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7340dff2-hpcmg\" (UID: \"680aaedd-5d27-4058-a444-b11825314d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" Nov 03 20:03:03.173626 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.173549 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680aaedd-5d27-4058-a444-b11825314d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7340dff2-hpcmg\" (UID: \"680aaedd-5d27-4058-a444-b11825314d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" Nov 03 20:03:03.174208 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.174185 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680aaedd-5d27-4058-a444-b11825314d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7340dff2-hpcmg\" (UID: \"680aaedd-5d27-4058-a444-b11825314d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" Nov 03 20:03:03.175948 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.175927 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680aaedd-5d27-4058-a444-b11825314d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7340dff2-hpcmg\" (UID: \"680aaedd-5d27-4058-a444-b11825314d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" Nov 03 20:03:03.181180 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.181160 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvvhb\" (UniqueName: \"kubernetes.io/projected/680aaedd-5d27-4058-a444-b11825314d65-kube-api-access-zvvhb\") pod \"maas-api-revoke-sub-7340dff2-hpcmg\" (UID: \"680aaedd-5d27-4058-a444-b11825314d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" Nov 03 20:03:03.276274 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.276248 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" Nov 03 20:03:03.276432 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.276283 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c" event={"ID":"734a8dba-3f4c-4c43-adea-556a98c0d057","Type":"ContainerDied","Data":"d2f8e835b670433b7f0abc7e236ba68165ccbe02a0d89b2e19541179ed6597a7"} Nov 03 20:03:03.276432 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.276331 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2f8e835b670433b7f0abc7e236ba68165ccbe02a0d89b2e19541179ed6597a7" Nov 03 20:03:03.277685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.277654 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t" event={"ID":"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da","Type":"ContainerStarted","Data":"85c9313763f860df311cea4f42011bb071057f51073413de4f22186fc050e546"} Nov 03 20:03:03.299680 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.299652 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn"] Nov 03 20:03:03.301096 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:03:03.301070 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7efe8399_943b_48ac_8555_72198abc6c9d.slice/crio-a53e3775b9f346ff80e4c7274a1a6a0091335b43b3b5e10f0067a852a0990d2d WatchSource:0}: Error finding container a53e3775b9f346ff80e4c7274a1a6a0091335b43b3b5e10f0067a852a0990d2d: Status 404 returned error can't find the container with id a53e3775b9f346ff80e4c7274a1a6a0091335b43b3b5e10f0067a852a0990d2d Nov 03 20:03:03.322162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.322136 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" Nov 03 20:03:03.580643 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.580612 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm"] Nov 03 20:03:03.584732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.584713 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm" Nov 03 20:03:03.593483 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.593451 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm"] Nov 03 20:03:03.610746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.610517 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg"] Nov 03 20:03:03.690616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.690571 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a347a031-2c7a-46c3-ba61-9825192b7a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad23aea1-bxkgm\" (UID: \"a347a031-2c7a-46c3-ba61-9825192b7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm" Nov 03 20:03:03.691136 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.691109 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a347a031-2c7a-46c3-ba61-9825192b7a9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad23aea1-bxkgm\" (UID: \"a347a031-2c7a-46c3-ba61-9825192b7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm" Nov 03 20:03:03.691392 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.691361 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nthwx\" (UniqueName: \"kubernetes.io/projected/a347a031-2c7a-46c3-ba61-9825192b7a9e-kube-api-access-nthwx\") pod \"maas-api-revoke-sub-ad23aea1-bxkgm\" (UID: \"a347a031-2c7a-46c3-ba61-9825192b7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm" Nov 03 20:03:03.792387 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.792352 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a347a031-2c7a-46c3-ba61-9825192b7a9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad23aea1-bxkgm\" (UID: \"a347a031-2c7a-46c3-ba61-9825192b7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm" Nov 03 20:03:03.792387 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.792390 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nthwx\" (UniqueName: \"kubernetes.io/projected/a347a031-2c7a-46c3-ba61-9825192b7a9e-kube-api-access-nthwx\") pod \"maas-api-revoke-sub-ad23aea1-bxkgm\" (UID: \"a347a031-2c7a-46c3-ba61-9825192b7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm" Nov 03 20:03:03.792600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.792475 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a347a031-2c7a-46c3-ba61-9825192b7a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad23aea1-bxkgm\" (UID: \"a347a031-2c7a-46c3-ba61-9825192b7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm" Nov 03 20:03:03.793111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.793083 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a347a031-2c7a-46c3-ba61-9825192b7a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad23aea1-bxkgm\" (UID: \"a347a031-2c7a-46c3-ba61-9825192b7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm" Nov 03 20:03:03.794627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.794599 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a347a031-2c7a-46c3-ba61-9825192b7a9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad23aea1-bxkgm\" (UID: \"a347a031-2c7a-46c3-ba61-9825192b7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm" Nov 03 20:03:03.800147 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.800129 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nthwx\" (UniqueName: \"kubernetes.io/projected/a347a031-2c7a-46c3-ba61-9825192b7a9e-kube-api-access-nthwx\") pod \"maas-api-revoke-sub-ad23aea1-bxkgm\" (UID: \"a347a031-2c7a-46c3-ba61-9825192b7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm" Nov 03 20:03:03.902384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:03.902352 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm" Nov 03 20:03:04.272637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:04.272604 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm"] Nov 03 20:03:04.285825 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:04.285797 5111 generic.go:356] "Generic (PLEG): container finished" podID="1c82d407-70a8-4f02-a62e-c3c6f0cbd3da" containerID="94940369510c9332d811c3a67cdf72a00cc696fdeae10b1bafba2b1fd6e27fa7" exitCode=0 Nov 03 20:03:04.285937 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:04.285882 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t" event={"ID":"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da","Type":"ContainerDied","Data":"94940369510c9332d811c3a67cdf72a00cc696fdeae10b1bafba2b1fd6e27fa7"} Nov 03 20:03:04.287721 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:04.287699 5111 generic.go:356] "Generic (PLEG): container finished" podID="7efe8399-943b-48ac-8555-72198abc6c9d" containerID="b86c97eaeee25478dec82ee77cbc6a6335702c97c722ba45205c24aad27187fb" exitCode=0 Nov 03 20:03:04.287846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:04.287787 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn" event={"ID":"7efe8399-943b-48ac-8555-72198abc6c9d","Type":"ContainerDied","Data":"b86c97eaeee25478dec82ee77cbc6a6335702c97c722ba45205c24aad27187fb"} Nov 03 20:03:04.287846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:04.287828 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn" event={"ID":"7efe8399-943b-48ac-8555-72198abc6c9d","Type":"ContainerStarted","Data":"a53e3775b9f346ff80e4c7274a1a6a0091335b43b3b5e10f0067a852a0990d2d"} Nov 03 20:03:04.289573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:04.289550 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" event={"ID":"680aaedd-5d27-4058-a444-b11825314d65","Type":"ContainerStarted","Data":"11adb005aaf0cb005283d94794ee746ac76b490b77f680eb8462e851c3509ec3"} Nov 03 20:03:04.289705 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:04.289590 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" event={"ID":"680aaedd-5d27-4058-a444-b11825314d65","Type":"ContainerStarted","Data":"71511665f7f99e35c51cfe4cd2af841405d6bd4df09c1573f71b69e35a668fa4"} Nov 03 20:03:04.292867 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:03:04.292836 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda347a031_2c7a_46c3_ba61_9825192b7a9e.slice/crio-b987f50584638dd2ad36fadefe454898b00c69afcb2026f221606744dcb105b2 WatchSource:0}: Error finding container b987f50584638dd2ad36fadefe454898b00c69afcb2026f221606744dcb105b2: Status 404 returned error can't find the container with id b987f50584638dd2ad36fadefe454898b00c69afcb2026f221606744dcb105b2 Nov 03 20:03:04.321509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:04.321461 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" podStartSLOduration=2.321447993 podStartE2EDuration="2.321447993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:04.318948654 +0000 UTC m=+1414.281327093" watchObservedRunningTime="2026-11-03 20:03:04.321447993 +0000 UTC m=+1414.283826432" Nov 03 20:03:05.061258 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:03:05.061164 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:03:05.298174 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.298141 5111 generic.go:356] "Generic (PLEG): container finished" podID="a347a031-2c7a-46c3-ba61-9825192b7a9e" containerID="9cc7269e0c250bb3221fa1355a13f61ef7fc07abfdc3f33b03752b30e1f30c6a" exitCode=0 Nov 03 20:03:05.298337 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.298222 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm" event={"ID":"a347a031-2c7a-46c3-ba61-9825192b7a9e","Type":"ContainerDied","Data":"9cc7269e0c250bb3221fa1355a13f61ef7fc07abfdc3f33b03752b30e1f30c6a"} Nov 03 20:03:05.298337 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.298286 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm" event={"ID":"a347a031-2c7a-46c3-ba61-9825192b7a9e","Type":"ContainerStarted","Data":"b987f50584638dd2ad36fadefe454898b00c69afcb2026f221606744dcb105b2"} Nov 03 20:03:05.299793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.299770 5111 generic.go:356] "Generic (PLEG): container finished" podID="680aaedd-5d27-4058-a444-b11825314d65" containerID="11adb005aaf0cb005283d94794ee746ac76b490b77f680eb8462e851c3509ec3" exitCode=0 Nov 03 20:03:05.299900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.299847 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" event={"ID":"680aaedd-5d27-4058-a444-b11825314d65","Type":"ContainerDied","Data":"11adb005aaf0cb005283d94794ee746ac76b490b77f680eb8462e851c3509ec3"} Nov 03 20:03:05.477069 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.477034 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t" Nov 03 20:03:05.519994 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.519963 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-maas-api-openshift-ca\") pod \"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da\" (UID: \"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da\") " Nov 03 20:03:05.520697 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.520674 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-kube-api-access-kbf2m\" (UniqueName: \"kubernetes.io/projected/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-kube-api-access-kbf2m\") pod \"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da\" (UID: \"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da\") " Nov 03 20:03:05.521473 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.520512 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-maas-api-openshift-ca" pod "1c82d407-70a8-4f02-a62e-c3c6f0cbd3da" (UID: "1c82d407-70a8-4f02-a62e-c3c6f0cbd3da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:05.521665 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.521647 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-maas-api-xks-ca\") pod \"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da\" (UID: \"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da\") " Nov 03 20:03:05.522554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.522517 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:05.526446 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.526412 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-kube-api-access-kbf2m" pod "1c82d407-70a8-4f02-a62e-c3c6f0cbd3da" (UID: "1c82d407-70a8-4f02-a62e-c3c6f0cbd3da"). InnerVolumeSpecName "kube-api-access-kbf2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:05.526675 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.526656 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-maas-api-xks-ca" pod "1c82d407-70a8-4f02-a62e-c3c6f0cbd3da" (UID: "1c82d407-70a8-4f02-a62e-c3c6f0cbd3da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:05.560511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.560478 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn" Nov 03 20:03:05.623461 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.623424 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7efe8399-943b-48ac-8555-72198abc6c9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efe8399-943b-48ac-8555-72198abc6c9d-maas-api-xks-ca\") pod \"7efe8399-943b-48ac-8555-72198abc6c9d\" (UID: \"7efe8399-943b-48ac-8555-72198abc6c9d\") " Nov 03 20:03:05.623619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.623528 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7efe8399-943b-48ac-8555-72198abc6c9d-kube-api-access-5pqcn\" (UniqueName: \"kubernetes.io/projected/7efe8399-943b-48ac-8555-72198abc6c9d-kube-api-access-5pqcn\") pod \"7efe8399-943b-48ac-8555-72198abc6c9d\" (UID: \"7efe8399-943b-48ac-8555-72198abc6c9d\") " Nov 03 20:03:05.623698 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.623629 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7efe8399-943b-48ac-8555-72198abc6c9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efe8399-943b-48ac-8555-72198abc6c9d-maas-api-openshift-ca\") pod \"7efe8399-943b-48ac-8555-72198abc6c9d\" (UID: \"7efe8399-943b-48ac-8555-72198abc6c9d\") " Nov 03 20:03:05.623993 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.623963 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efe8399-943b-48ac-8555-72198abc6c9d-maas-api-openshift-ca" pod "7efe8399-943b-48ac-8555-72198abc6c9d" (UID: "7efe8399-943b-48ac-8555-72198abc6c9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:05.624289 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.624267 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:05.624381 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.624295 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbf2m\" (UniqueName: \"kubernetes.io/projected/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da-kube-api-access-kbf2m\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:05.624381 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.624311 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efe8399-943b-48ac-8555-72198abc6c9d-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:05.625780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.625757 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7efe8399-943b-48ac-8555-72198abc6c9d-kube-api-access-5pqcn" pod "7efe8399-943b-48ac-8555-72198abc6c9d" (UID: "7efe8399-943b-48ac-8555-72198abc6c9d"). InnerVolumeSpecName "kube-api-access-5pqcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:05.625780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.625774 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efe8399-943b-48ac-8555-72198abc6c9d-maas-api-xks-ca" pod "7efe8399-943b-48ac-8555-72198abc6c9d" (UID: "7efe8399-943b-48ac-8555-72198abc6c9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:05.725011 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.724976 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efe8399-943b-48ac-8555-72198abc6c9d-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:05.725011 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:05.725005 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pqcn\" (UniqueName: \"kubernetes.io/projected/7efe8399-943b-48ac-8555-72198abc6c9d-kube-api-access-5pqcn\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:06.308677 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.308641 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t" Nov 03 20:03:06.309175 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.308640 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t" event={"ID":"1c82d407-70a8-4f02-a62e-c3c6f0cbd3da","Type":"ContainerDied","Data":"85c9313763f860df311cea4f42011bb071057f51073413de4f22186fc050e546"} Nov 03 20:03:06.309175 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.308775 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85c9313763f860df311cea4f42011bb071057f51073413de4f22186fc050e546" Nov 03 20:03:06.310116 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.310101 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn" Nov 03 20:03:06.310209 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.310154 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn" event={"ID":"7efe8399-943b-48ac-8555-72198abc6c9d","Type":"ContainerDied","Data":"a53e3775b9f346ff80e4c7274a1a6a0091335b43b3b5e10f0067a852a0990d2d"} Nov 03 20:03:06.310209 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.310199 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a53e3775b9f346ff80e4c7274a1a6a0091335b43b3b5e10f0067a852a0990d2d" Nov 03 20:03:06.494214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.494194 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" Nov 03 20:03:06.525503 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.525471 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm" Nov 03 20:03:06.534242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.534213 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680aaedd-5d27-4058-a444-b11825314d65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680aaedd-5d27-4058-a444-b11825314d65-maas-api-xks-ca\") pod \"680aaedd-5d27-4058-a444-b11825314d65\" (UID: \"680aaedd-5d27-4058-a444-b11825314d65\") " Nov 03 20:03:06.534378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.534293 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680aaedd-5d27-4058-a444-b11825314d65-kube-api-access-zvvhb\" (UniqueName: \"kubernetes.io/projected/680aaedd-5d27-4058-a444-b11825314d65-kube-api-access-zvvhb\") pod \"680aaedd-5d27-4058-a444-b11825314d65\" (UID: \"680aaedd-5d27-4058-a444-b11825314d65\") " Nov 03 20:03:06.534889 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.534462 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/680aaedd-5d27-4058-a444-b11825314d65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680aaedd-5d27-4058-a444-b11825314d65-maas-api-openshift-ca\") pod \"680aaedd-5d27-4058-a444-b11825314d65\" (UID: \"680aaedd-5d27-4058-a444-b11825314d65\") " Nov 03 20:03:06.534889 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.534852 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680aaedd-5d27-4058-a444-b11825314d65-maas-api-openshift-ca" pod "680aaedd-5d27-4058-a444-b11825314d65" (UID: "680aaedd-5d27-4058-a444-b11825314d65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:06.535141 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.535112 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680aaedd-5d27-4058-a444-b11825314d65-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:06.536974 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.536950 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680aaedd-5d27-4058-a444-b11825314d65-maas-api-xks-ca" pod "680aaedd-5d27-4058-a444-b11825314d65" (UID: "680aaedd-5d27-4058-a444-b11825314d65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:06.537085 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.536980 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680aaedd-5d27-4058-a444-b11825314d65-kube-api-access-zvvhb" pod "680aaedd-5d27-4058-a444-b11825314d65" (UID: "680aaedd-5d27-4058-a444-b11825314d65"). InnerVolumeSpecName "kube-api-access-zvvhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:06.636275 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.636244 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a347a031-2c7a-46c3-ba61-9825192b7a9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a347a031-2c7a-46c3-ba61-9825192b7a9e-maas-api-openshift-ca\") pod \"a347a031-2c7a-46c3-ba61-9825192b7a9e\" (UID: \"a347a031-2c7a-46c3-ba61-9825192b7a9e\") " Nov 03 20:03:06.636421 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.636287 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a347a031-2c7a-46c3-ba61-9825192b7a9e-kube-api-access-nthwx\" (UniqueName: \"kubernetes.io/projected/a347a031-2c7a-46c3-ba61-9825192b7a9e-kube-api-access-nthwx\") pod \"a347a031-2c7a-46c3-ba61-9825192b7a9e\" (UID: \"a347a031-2c7a-46c3-ba61-9825192b7a9e\") " Nov 03 20:03:06.636421 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.636360 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a347a031-2c7a-46c3-ba61-9825192b7a9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a347a031-2c7a-46c3-ba61-9825192b7a9e-maas-api-xks-ca\") pod \"a347a031-2c7a-46c3-ba61-9825192b7a9e\" (UID: \"a347a031-2c7a-46c3-ba61-9825192b7a9e\") " Nov 03 20:03:06.636689 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.636666 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a347a031-2c7a-46c3-ba61-9825192b7a9e-maas-api-openshift-ca" pod "a347a031-2c7a-46c3-ba61-9825192b7a9e" (UID: "a347a031-2c7a-46c3-ba61-9825192b7a9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:06.636777 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.636727 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680aaedd-5d27-4058-a444-b11825314d65-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:06.636777 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.636741 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvvhb\" (UniqueName: \"kubernetes.io/projected/680aaedd-5d27-4058-a444-b11825314d65-kube-api-access-zvvhb\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:06.638450 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.638425 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a347a031-2c7a-46c3-ba61-9825192b7a9e-kube-api-access-nthwx" pod "a347a031-2c7a-46c3-ba61-9825192b7a9e" (UID: "a347a031-2c7a-46c3-ba61-9825192b7a9e"). InnerVolumeSpecName "kube-api-access-nthwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:06.638601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.638470 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a347a031-2c7a-46c3-ba61-9825192b7a9e-maas-api-xks-ca" pod "a347a031-2c7a-46c3-ba61-9825192b7a9e" (UID: "a347a031-2c7a-46c3-ba61-9825192b7a9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:06.740495 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.738716 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a347a031-2c7a-46c3-ba61-9825192b7a9e-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:06.740495 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.738752 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nthwx\" (UniqueName: \"kubernetes.io/projected/a347a031-2c7a-46c3-ba61-9825192b7a9e-kube-api-access-nthwx\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:06.740495 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:06.738766 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a347a031-2c7a-46c3-ba61-9825192b7a9e-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:07.318560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.318510 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" Nov 03 20:03:07.318992 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.318560 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg" event={"ID":"680aaedd-5d27-4058-a444-b11825314d65","Type":"ContainerDied","Data":"71511665f7f99e35c51cfe4cd2af841405d6bd4df09c1573f71b69e35a668fa4"} Nov 03 20:03:07.318992 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.318622 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71511665f7f99e35c51cfe4cd2af841405d6bd4df09c1573f71b69e35a668fa4" Nov 03 20:03:07.320198 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.320164 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm" event={"ID":"a347a031-2c7a-46c3-ba61-9825192b7a9e","Type":"ContainerDied","Data":"b987f50584638dd2ad36fadefe454898b00c69afcb2026f221606744dcb105b2"} Nov 03 20:03:07.320323 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.320214 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b987f50584638dd2ad36fadefe454898b00c69afcb2026f221606744dcb105b2" Nov 03 20:03:07.320323 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.320263 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm" Nov 03 20:03:07.474951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.474919 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj"] Nov 03 20:03:07.475783 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.475761 5111 cpu_manager.go:414] "containerMap: removing container" podUID="7efe8399-943b-48ac-8555-72198abc6c9d" containerName="revoke-keys" Nov 03 20:03:07.475783 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.475783 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7efe8399-943b-48ac-8555-72198abc6c9d" containerName="revoke-keys" Nov 03 20:03:07.475957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.475798 5111 cpu_manager.go:414] "containerMap: removing container" podUID="1c82d407-70a8-4f02-a62e-c3c6f0cbd3da" containerName="revoke-keys" Nov 03 20:03:07.475957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.475804 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c82d407-70a8-4f02-a62e-c3c6f0cbd3da" containerName="revoke-keys" Nov 03 20:03:07.475957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.475813 5111 cpu_manager.go:414] "containerMap: removing container" podUID="a347a031-2c7a-46c3-ba61-9825192b7a9e" containerName="revoke-keys" Nov 03 20:03:07.475957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.475819 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a347a031-2c7a-46c3-ba61-9825192b7a9e" containerName="revoke-keys" Nov 03 20:03:07.475957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.475828 5111 cpu_manager.go:414] "containerMap: removing container" podUID="680aaedd-5d27-4058-a444-b11825314d65" containerName="revoke-keys" Nov 03 20:03:07.475957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.475833 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680aaedd-5d27-4058-a444-b11825314d65" containerName="revoke-keys" Nov 03 20:03:07.475957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.475927 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="7efe8399-943b-48ac-8555-72198abc6c9d" containerName="revoke-keys" Nov 03 20:03:07.475957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.475936 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c82d407-70a8-4f02-a62e-c3c6f0cbd3da" containerName="revoke-keys" Nov 03 20:03:07.475957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.475942 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="a347a031-2c7a-46c3-ba61-9825192b7a9e" containerName="revoke-keys" Nov 03 20:03:07.475957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.475950 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="680aaedd-5d27-4058-a444-b11825314d65" containerName="revoke-keys" Nov 03 20:03:07.480217 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.480194 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj" Nov 03 20:03:07.484072 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.484046 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj"] Nov 03 20:03:07.484369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.484348 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:03:07.485597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.484426 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:03:07.505988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.505966 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:03:07.506290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.506258 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:03:07.554418 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.554392 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d879f60-3aa2-4545-b235-b38eb1b09b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5afefd0-lq5wj\" (UID: \"6d879f60-3aa2-4545-b235-b38eb1b09b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj" Nov 03 20:03:07.554584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.554432 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d879f60-3aa2-4545-b235-b38eb1b09b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5afefd0-lq5wj\" (UID: \"6d879f60-3aa2-4545-b235-b38eb1b09b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj" Nov 03 20:03:07.554664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.554587 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrc4h\" (UniqueName: \"kubernetes.io/projected/6d879f60-3aa2-4545-b235-b38eb1b09b96-kube-api-access-lrc4h\") pod \"maas-api-revoke-sub-f5afefd0-lq5wj\" (UID: \"6d879f60-3aa2-4545-b235-b38eb1b09b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj" Nov 03 20:03:07.655834 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.655800 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d879f60-3aa2-4545-b235-b38eb1b09b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5afefd0-lq5wj\" (UID: \"6d879f60-3aa2-4545-b235-b38eb1b09b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj" Nov 03 20:03:07.655960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.655849 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d879f60-3aa2-4545-b235-b38eb1b09b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5afefd0-lq5wj\" (UID: \"6d879f60-3aa2-4545-b235-b38eb1b09b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj" Nov 03 20:03:07.655960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.655926 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrc4h\" (UniqueName: \"kubernetes.io/projected/6d879f60-3aa2-4545-b235-b38eb1b09b96-kube-api-access-lrc4h\") pod \"maas-api-revoke-sub-f5afefd0-lq5wj\" (UID: \"6d879f60-3aa2-4545-b235-b38eb1b09b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj" Nov 03 20:03:07.656600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.656528 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d879f60-3aa2-4545-b235-b38eb1b09b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5afefd0-lq5wj\" (UID: \"6d879f60-3aa2-4545-b235-b38eb1b09b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj" Nov 03 20:03:07.658100 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.658082 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d879f60-3aa2-4545-b235-b38eb1b09b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5afefd0-lq5wj\" (UID: \"6d879f60-3aa2-4545-b235-b38eb1b09b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj" Nov 03 20:03:07.663688 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.663669 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrc4h\" (UniqueName: \"kubernetes.io/projected/6d879f60-3aa2-4545-b235-b38eb1b09b96-kube-api-access-lrc4h\") pod \"maas-api-revoke-sub-f5afefd0-lq5wj\" (UID: \"6d879f60-3aa2-4545-b235-b38eb1b09b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj" Nov 03 20:03:07.801481 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:07.801451 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj" Nov 03 20:03:08.085335 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:08.085305 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj"] Nov 03 20:03:08.086299 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:03:08.086273 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d879f60_3aa2_4545_b235_b38eb1b09b96.slice/crio-e1415f69dd5f440af0a494159edc0b2238d3c4ab15b950d78bf5cd77351a2166 WatchSource:0}: Error finding container e1415f69dd5f440af0a494159edc0b2238d3c4ab15b950d78bf5cd77351a2166: Status 404 returned error can't find the container with id e1415f69dd5f440af0a494159edc0b2238d3c4ab15b950d78bf5cd77351a2166 Nov 03 20:03:08.327542 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:08.327502 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj" event={"ID":"6d879f60-3aa2-4545-b235-b38eb1b09b96","Type":"ContainerStarted","Data":"e1415f69dd5f440af0a494159edc0b2238d3c4ab15b950d78bf5cd77351a2166"} Nov 03 20:03:09.366557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:09.366426 5111 generic.go:356] "Generic (PLEG): container finished" podID="6d879f60-3aa2-4545-b235-b38eb1b09b96" containerID="1e793ad93b7e6ee16a65000aaeae957d61caa31cf9da8a30f70d9d9ccab59505" exitCode=0 Nov 03 20:03:09.366557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:09.366551 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj" event={"ID":"6d879f60-3aa2-4545-b235-b38eb1b09b96","Type":"ContainerDied","Data":"1e793ad93b7e6ee16a65000aaeae957d61caa31cf9da8a30f70d9d9ccab59505"} Nov 03 20:03:10.392446 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:10.392391 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:10.439674 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:10.437850 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:10.451621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:10.450627 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 20:03:10.514567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:10.513853 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:10.831557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:10.827698 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj" Nov 03 20:03:11.027575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:11.027463 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d879f60-3aa2-4545-b235-b38eb1b09b96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d879f60-3aa2-4545-b235-b38eb1b09b96-maas-api-openshift-ca\") pod \"6d879f60-3aa2-4545-b235-b38eb1b09b96\" (UID: \"6d879f60-3aa2-4545-b235-b38eb1b09b96\") " Nov 03 20:03:11.027836 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:11.027814 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d879f60-3aa2-4545-b235-b38eb1b09b96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d879f60-3aa2-4545-b235-b38eb1b09b96-maas-api-xks-ca\") pod \"6d879f60-3aa2-4545-b235-b38eb1b09b96\" (UID: \"6d879f60-3aa2-4545-b235-b38eb1b09b96\") " Nov 03 20:03:11.042890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:11.027998 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d879f60-3aa2-4545-b235-b38eb1b09b96-kube-api-access-lrc4h\" (UniqueName: \"kubernetes.io/projected/6d879f60-3aa2-4545-b235-b38eb1b09b96-kube-api-access-lrc4h\") pod \"6d879f60-3aa2-4545-b235-b38eb1b09b96\" (UID: \"6d879f60-3aa2-4545-b235-b38eb1b09b96\") " Nov 03 20:03:11.104567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:11.101781 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d879f60-3aa2-4545-b235-b38eb1b09b96-maas-api-openshift-ca" pod "6d879f60-3aa2-4545-b235-b38eb1b09b96" (UID: "6d879f60-3aa2-4545-b235-b38eb1b09b96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:11.160900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:11.160865 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d879f60-3aa2-4545-b235-b38eb1b09b96-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:11.286612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:11.282716 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d879f60-3aa2-4545-b235-b38eb1b09b96-kube-api-access-lrc4h" pod "6d879f60-3aa2-4545-b235-b38eb1b09b96" (UID: "6d879f60-3aa2-4545-b235-b38eb1b09b96"). InnerVolumeSpecName "kube-api-access-lrc4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:11.321204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:11.321163 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d879f60-3aa2-4545-b235-b38eb1b09b96-maas-api-xks-ca" pod "6d879f60-3aa2-4545-b235-b38eb1b09b96" (UID: "6d879f60-3aa2-4545-b235-b38eb1b09b96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:11.366754 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:11.366716 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d879f60-3aa2-4545-b235-b38eb1b09b96-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:11.366993 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:11.366973 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrc4h\" (UniqueName: \"kubernetes.io/projected/6d879f60-3aa2-4545-b235-b38eb1b09b96-kube-api-access-lrc4h\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:11.470856 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:11.470815 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj" event={"ID":"6d879f60-3aa2-4545-b235-b38eb1b09b96","Type":"ContainerDied","Data":"e1415f69dd5f440af0a494159edc0b2238d3c4ab15b950d78bf5cd77351a2166"} Nov 03 20:03:11.471477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:11.471455 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1415f69dd5f440af0a494159edc0b2238d3c4ab15b950d78bf5cd77351a2166" Nov 03 20:03:11.475723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:11.475702 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj" Nov 03 20:03:12.487556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:12.486674 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:12.639389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:12.639344 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:15.345508 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.345465 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr"] Nov 03 20:03:15.477084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.476665 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6d879f60-3aa2-4545-b235-b38eb1b09b96" containerName="revoke-keys" Nov 03 20:03:15.477278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.477261 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d879f60-3aa2-4545-b235-b38eb1b09b96" containerName="revoke-keys" Nov 03 20:03:15.477460 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.477451 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d879f60-3aa2-4545-b235-b38eb1b09b96" containerName="revoke-keys" Nov 03 20:03:15.483168 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.483144 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr"] Nov 03 20:03:15.483413 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.483396 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr" Nov 03 20:03:15.488910 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.488888 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:03:15.489255 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.489238 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:03:15.489982 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.489963 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:03:15.490176 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.490161 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:03:15.490290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.490278 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6"] Nov 03 20:03:15.495109 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.495095 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6"] Nov 03 20:03:15.495267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.495256 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6" Nov 03 20:03:15.594356 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.594312 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bfd44b2-2khxr\" (UID: \"5f61d390-7b4f-4bc5-a77e-3ea7541f312a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr" Nov 03 20:03:15.594727 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.594697 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2da28f1-2hfc6\" (UID: \"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6" Nov 03 20:03:15.594835 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.594782 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bfd44b2-2khxr\" (UID: \"5f61d390-7b4f-4bc5-a77e-3ea7541f312a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr" Nov 03 20:03:15.594892 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.594833 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2da28f1-2hfc6\" (UID: \"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6" Nov 03 20:03:15.594957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.594889 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpg9l\" (UniqueName: \"kubernetes.io/projected/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-kube-api-access-xpg9l\") pod \"maas-api-revoke-sub-c2da28f1-2hfc6\" (UID: \"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6" Nov 03 20:03:15.595017 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.594974 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqls4\" (UniqueName: \"kubernetes.io/projected/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-kube-api-access-lqls4\") pod \"maas-api-revoke-sub-4bfd44b2-2khxr\" (UID: \"5f61d390-7b4f-4bc5-a77e-3ea7541f312a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr" Nov 03 20:03:15.696391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.696352 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqls4\" (UniqueName: \"kubernetes.io/projected/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-kube-api-access-lqls4\") pod \"maas-api-revoke-sub-4bfd44b2-2khxr\" (UID: \"5f61d390-7b4f-4bc5-a77e-3ea7541f312a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr" Nov 03 20:03:15.696599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.696469 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bfd44b2-2khxr\" (UID: \"5f61d390-7b4f-4bc5-a77e-3ea7541f312a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr" Nov 03 20:03:15.696681 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.696601 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2da28f1-2hfc6\" (UID: \"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6" Nov 03 20:03:15.697132 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.697105 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bfd44b2-2khxr\" (UID: \"5f61d390-7b4f-4bc5-a77e-3ea7541f312a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr" Nov 03 20:03:15.697391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.697362 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2da28f1-2hfc6\" (UID: \"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6" Nov 03 20:03:15.697498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.697436 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpg9l\" (UniqueName: \"kubernetes.io/projected/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-kube-api-access-xpg9l\") pod \"maas-api-revoke-sub-c2da28f1-2hfc6\" (UID: \"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6" Nov 03 20:03:15.697853 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.697831 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bfd44b2-2khxr\" (UID: \"5f61d390-7b4f-4bc5-a77e-3ea7541f312a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr" Nov 03 20:03:15.698486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.698460 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2da28f1-2hfc6\" (UID: \"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6" Nov 03 20:03:15.699493 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.699471 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2da28f1-2hfc6\" (UID: \"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6" Nov 03 20:03:15.699827 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.699809 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bfd44b2-2khxr\" (UID: \"5f61d390-7b4f-4bc5-a77e-3ea7541f312a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr" Nov 03 20:03:15.705141 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.705092 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpg9l\" (UniqueName: \"kubernetes.io/projected/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-kube-api-access-xpg9l\") pod \"maas-api-revoke-sub-c2da28f1-2hfc6\" (UID: \"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6" Nov 03 20:03:15.705348 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.705329 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqls4\" (UniqueName: \"kubernetes.io/projected/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-kube-api-access-lqls4\") pod \"maas-api-revoke-sub-4bfd44b2-2khxr\" (UID: \"5f61d390-7b4f-4bc5-a77e-3ea7541f312a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr" Nov 03 20:03:15.789329 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.789296 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7"] Nov 03 20:03:15.795669 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.795638 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" Nov 03 20:03:15.804034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.804003 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7"] Nov 03 20:03:15.827850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.827817 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr" Nov 03 20:03:15.842635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.842609 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6" Nov 03 20:03:15.939193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.939149 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4"] Nov 03 20:03:15.952433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.952398 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8f93ca-515a-4785-98e1-1830339ca556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a710ef4-269v7\" (UID: \"3c8f93ca-515a-4785-98e1-1830339ca556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" Nov 03 20:03:15.952588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.952576 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8f93ca-515a-4785-98e1-1830339ca556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a710ef4-269v7\" (UID: \"3c8f93ca-515a-4785-98e1-1830339ca556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" Nov 03 20:03:15.952749 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.952683 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znc6x\" (UniqueName: \"kubernetes.io/projected/3c8f93ca-515a-4785-98e1-1830339ca556-kube-api-access-znc6x\") pod \"maas-api-revoke-sub-1a710ef4-269v7\" (UID: \"3c8f93ca-515a-4785-98e1-1830339ca556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" Nov 03 20:03:15.983343 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.982720 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4"] Nov 03 20:03:15.983343 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:15.982845 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" Nov 03 20:03:16.054485 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.054452 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8f93ca-515a-4785-98e1-1830339ca556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a710ef4-269v7\" (UID: \"3c8f93ca-515a-4785-98e1-1830339ca556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" Nov 03 20:03:16.054683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.054498 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znc6x\" (UniqueName: \"kubernetes.io/projected/3c8f93ca-515a-4785-98e1-1830339ca556-kube-api-access-znc6x\") pod \"maas-api-revoke-sub-1a710ef4-269v7\" (UID: \"3c8f93ca-515a-4785-98e1-1830339ca556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" Nov 03 20:03:16.054683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.054667 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8f93ca-515a-4785-98e1-1830339ca556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a710ef4-269v7\" (UID: \"3c8f93ca-515a-4785-98e1-1830339ca556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" Nov 03 20:03:16.055240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.055219 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8f93ca-515a-4785-98e1-1830339ca556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a710ef4-269v7\" (UID: \"3c8f93ca-515a-4785-98e1-1830339ca556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" Nov 03 20:03:16.057054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.057031 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8f93ca-515a-4785-98e1-1830339ca556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a710ef4-269v7\" (UID: \"3c8f93ca-515a-4785-98e1-1830339ca556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" Nov 03 20:03:16.064991 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.064964 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znc6x\" (UniqueName: \"kubernetes.io/projected/3c8f93ca-515a-4785-98e1-1830339ca556-kube-api-access-znc6x\") pod \"maas-api-revoke-sub-1a710ef4-269v7\" (UID: \"3c8f93ca-515a-4785-98e1-1830339ca556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" Nov 03 20:03:16.121987 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.121960 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr"] Nov 03 20:03:16.122479 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.122464 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" Nov 03 20:03:16.122851 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:03:16.122825 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f61d390_7b4f_4bc5_a77e_3ea7541f312a.slice/crio-91deed5f18b97472bae04d90db20451d8ce9f86cb2bd46047fb76ace49260534 WatchSource:0}: Error finding container 91deed5f18b97472bae04d90db20451d8ce9f86cb2bd46047fb76ace49260534: Status 404 returned error can't find the container with id 91deed5f18b97472bae04d90db20451d8ce9f86cb2bd46047fb76ace49260534 Nov 03 20:03:16.156063 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.156024 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff5kq\" (UniqueName: \"kubernetes.io/projected/2afb37da-7177-4752-93ba-9a68a15a00c8-kube-api-access-ff5kq\") pod \"maas-api-revoke-sub-8d990d13-6grz4\" (UID: \"2afb37da-7177-4752-93ba-9a68a15a00c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" Nov 03 20:03:16.156281 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.156257 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2afb37da-7177-4752-93ba-9a68a15a00c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d990d13-6grz4\" (UID: \"2afb37da-7177-4752-93ba-9a68a15a00c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" Nov 03 20:03:16.156334 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.156302 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2afb37da-7177-4752-93ba-9a68a15a00c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d990d13-6grz4\" (UID: \"2afb37da-7177-4752-93ba-9a68a15a00c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" Nov 03 20:03:16.188722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.188630 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6"] Nov 03 20:03:16.258218 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.258181 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2afb37da-7177-4752-93ba-9a68a15a00c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d990d13-6grz4\" (UID: \"2afb37da-7177-4752-93ba-9a68a15a00c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" Nov 03 20:03:16.258393 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.258237 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2afb37da-7177-4752-93ba-9a68a15a00c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d990d13-6grz4\" (UID: \"2afb37da-7177-4752-93ba-9a68a15a00c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" Nov 03 20:03:16.258479 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.258448 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff5kq\" (UniqueName: \"kubernetes.io/projected/2afb37da-7177-4752-93ba-9a68a15a00c8-kube-api-access-ff5kq\") pod \"maas-api-revoke-sub-8d990d13-6grz4\" (UID: \"2afb37da-7177-4752-93ba-9a68a15a00c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" Nov 03 20:03:16.259266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.259243 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2afb37da-7177-4752-93ba-9a68a15a00c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d990d13-6grz4\" (UID: \"2afb37da-7177-4752-93ba-9a68a15a00c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" Nov 03 20:03:16.260795 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.260774 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2afb37da-7177-4752-93ba-9a68a15a00c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d990d13-6grz4\" (UID: \"2afb37da-7177-4752-93ba-9a68a15a00c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" Nov 03 20:03:16.266407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.266384 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff5kq\" (UniqueName: \"kubernetes.io/projected/2afb37da-7177-4752-93ba-9a68a15a00c8-kube-api-access-ff5kq\") pod \"maas-api-revoke-sub-8d990d13-6grz4\" (UID: \"2afb37da-7177-4752-93ba-9a68a15a00c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" Nov 03 20:03:16.316887 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.316854 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" Nov 03 20:03:16.422979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.422949 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7"] Nov 03 20:03:16.423440 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:03:16.423229 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8f93ca_515a_4785_98e1_1830339ca556.slice/crio-369c06a341e8d7d963f560b5e7298ac1a7cd99b42e424b184ce72ac9f867484a WatchSource:0}: Error finding container 369c06a341e8d7d963f560b5e7298ac1a7cd99b42e424b184ce72ac9f867484a: Status 404 returned error can't find the container with id 369c06a341e8d7d963f560b5e7298ac1a7cd99b42e424b184ce72ac9f867484a Nov 03 20:03:16.553041 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.553003 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr" event={"ID":"5f61d390-7b4f-4bc5-a77e-3ea7541f312a","Type":"ContainerStarted","Data":"91deed5f18b97472bae04d90db20451d8ce9f86cb2bd46047fb76ace49260534"} Nov 03 20:03:16.554426 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.554400 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6" event={"ID":"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4","Type":"ContainerStarted","Data":"bbee7156067b53d2a34bbe658106499ddd3793a377324dea46c4634469b8f522"} Nov 03 20:03:16.555935 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.555912 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" event={"ID":"3c8f93ca-515a-4785-98e1-1830339ca556","Type":"ContainerStarted","Data":"369c06a341e8d7d963f560b5e7298ac1a7cd99b42e424b184ce72ac9f867484a"} Nov 03 20:03:16.599968 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:16.599940 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4"] Nov 03 20:03:16.647052 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:03:16.647007 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2afb37da_7177_4752_93ba_9a68a15a00c8.slice/crio-badb0b270e75d8baa881b3b22bb6220562e160e323cb984db300ddafffebc40e WatchSource:0}: Error finding container badb0b270e75d8baa881b3b22bb6220562e160e323cb984db300ddafffebc40e: Status 404 returned error can't find the container with id badb0b270e75d8baa881b3b22bb6220562e160e323cb984db300ddafffebc40e Nov 03 20:03:17.566702 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:17.566665 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" event={"ID":"3c8f93ca-515a-4785-98e1-1830339ca556","Type":"ContainerStarted","Data":"582fe2e7839287c561417bc6f4849b9cdba041c406760c819b5a7a89b431071d"} Nov 03 20:03:17.568467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:17.568441 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" event={"ID":"2afb37da-7177-4752-93ba-9a68a15a00c8","Type":"ContainerStarted","Data":"06d6fa60fa9803a16aae4956a95a6d7ee1c6e1251376bc3b1c06a0c46f191e21"} Nov 03 20:03:17.568659 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:17.568488 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" event={"ID":"2afb37da-7177-4752-93ba-9a68a15a00c8","Type":"ContainerStarted","Data":"badb0b270e75d8baa881b3b22bb6220562e160e323cb984db300ddafffebc40e"} Nov 03 20:03:17.570264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:17.570241 5111 generic.go:356] "Generic (PLEG): container finished" podID="5f61d390-7b4f-4bc5-a77e-3ea7541f312a" containerID="0952d82be61664a23bcecd5bd289bd2be61ce5cbacfcc596b1968be9ee794dd0" exitCode=0 Nov 03 20:03:17.570358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:17.570314 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr" event={"ID":"5f61d390-7b4f-4bc5-a77e-3ea7541f312a","Type":"ContainerDied","Data":"0952d82be61664a23bcecd5bd289bd2be61ce5cbacfcc596b1968be9ee794dd0"} Nov 03 20:03:17.572967 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:17.572940 5111 generic.go:356] "Generic (PLEG): container finished" podID="154e9e64-a6bd-49f4-8b65-6c6ad5541fe4" containerID="50dec5ac6600d2651005dfb9e8db5e692901ad5b364c530e302b80a41f76c7d9" exitCode=0 Nov 03 20:03:17.573058 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:17.573019 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6" event={"ID":"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4","Type":"ContainerDied","Data":"50dec5ac6600d2651005dfb9e8db5e692901ad5b364c530e302b80a41f76c7d9"} Nov 03 20:03:17.622377 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:17.622326 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" podStartSLOduration=2.622311331 podStartE2EDuration="2.622311331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:17.618279576 +0000 UTC m=+1427.580658016" watchObservedRunningTime="2026-11-03 20:03:17.622311331 +0000 UTC m=+1427.584689769" Nov 03 20:03:18.581715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:18.581681 5111 generic.go:356] "Generic (PLEG): container finished" podID="3c8f93ca-515a-4785-98e1-1830339ca556" containerID="582fe2e7839287c561417bc6f4849b9cdba041c406760c819b5a7a89b431071d" exitCode=0 Nov 03 20:03:18.582211 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:18.581760 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" event={"ID":"3c8f93ca-515a-4785-98e1-1830339ca556","Type":"ContainerDied","Data":"582fe2e7839287c561417bc6f4849b9cdba041c406760c819b5a7a89b431071d"} Nov 03 20:03:18.583559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:18.583515 5111 generic.go:356] "Generic (PLEG): container finished" podID="2afb37da-7177-4752-93ba-9a68a15a00c8" containerID="06d6fa60fa9803a16aae4956a95a6d7ee1c6e1251376bc3b1c06a0c46f191e21" exitCode=0 Nov 03 20:03:18.583684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:18.583613 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" event={"ID":"2afb37da-7177-4752-93ba-9a68a15a00c8","Type":"ContainerDied","Data":"06d6fa60fa9803a16aae4956a95a6d7ee1c6e1251376bc3b1c06a0c46f191e21"} Nov 03 20:03:18.868283 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:18.868255 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6" Nov 03 20:03:18.871773 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:18.871748 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" Nov 03 20:03:18.876462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:18.876441 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr" Nov 03 20:03:19.016082 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.016051 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-maas-api-xks-ca\") pod \"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4\" (UID: \"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4\") " Nov 03 20:03:19.016082 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.016088 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8f93ca-515a-4785-98e1-1830339ca556-kube-api-access-znc6x\" (UniqueName: \"kubernetes.io/projected/3c8f93ca-515a-4785-98e1-1830339ca556-kube-api-access-znc6x\") pod \"3c8f93ca-515a-4785-98e1-1830339ca556\" (UID: \"3c8f93ca-515a-4785-98e1-1830339ca556\") " Nov 03 20:03:19.016294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.016183 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-maas-api-xks-ca\") pod \"5f61d390-7b4f-4bc5-a77e-3ea7541f312a\" (UID: \"5f61d390-7b4f-4bc5-a77e-3ea7541f312a\") " Nov 03 20:03:19.016294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.016245 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-kube-api-access-lqls4\" (UniqueName: \"kubernetes.io/projected/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-kube-api-access-lqls4\") pod \"5f61d390-7b4f-4bc5-a77e-3ea7541f312a\" (UID: \"5f61d390-7b4f-4bc5-a77e-3ea7541f312a\") " Nov 03 20:03:19.016294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.016286 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-maas-api-openshift-ca\") pod \"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4\" (UID: \"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4\") " Nov 03 20:03:19.016439 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.016413 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-kube-api-access-xpg9l\" (UniqueName: \"kubernetes.io/projected/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-kube-api-access-xpg9l\") pod \"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4\" (UID: \"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4\") " Nov 03 20:03:19.016498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.016486 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-maas-api-openshift-ca\") pod \"5f61d390-7b4f-4bc5-a77e-3ea7541f312a\" (UID: \"5f61d390-7b4f-4bc5-a77e-3ea7541f312a\") " Nov 03 20:03:19.016637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.016614 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c8f93ca-515a-4785-98e1-1830339ca556-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8f93ca-515a-4785-98e1-1830339ca556-maas-api-xks-ca\") pod \"3c8f93ca-515a-4785-98e1-1830339ca556\" (UID: \"3c8f93ca-515a-4785-98e1-1830339ca556\") " Nov 03 20:03:19.016775 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.016653 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c8f93ca-515a-4785-98e1-1830339ca556-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8f93ca-515a-4785-98e1-1830339ca556-maas-api-openshift-ca\") pod \"3c8f93ca-515a-4785-98e1-1830339ca556\" (UID: \"3c8f93ca-515a-4785-98e1-1830339ca556\") " Nov 03 20:03:19.016775 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.016694 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-maas-api-openshift-ca" pod "154e9e64-a6bd-49f4-8b65-6c6ad5541fe4" (UID: "154e9e64-a6bd-49f4-8b65-6c6ad5541fe4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:19.016990 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.016963 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-maas-api-openshift-ca" pod "5f61d390-7b4f-4bc5-a77e-3ea7541f312a" (UID: "5f61d390-7b4f-4bc5-a77e-3ea7541f312a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:19.017274 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.017254 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8f93ca-515a-4785-98e1-1830339ca556-maas-api-openshift-ca" pod "3c8f93ca-515a-4785-98e1-1830339ca556" (UID: "3c8f93ca-515a-4785-98e1-1830339ca556"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:19.017391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.017369 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8f93ca-515a-4785-98e1-1830339ca556-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:19.017455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.017400 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:19.017455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.017418 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:19.018773 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.018729 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-kube-api-access-xpg9l" pod "154e9e64-a6bd-49f4-8b65-6c6ad5541fe4" (UID: "154e9e64-a6bd-49f4-8b65-6c6ad5541fe4"). InnerVolumeSpecName "kube-api-access-xpg9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:19.018853 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.018809 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-kube-api-access-lqls4" pod "5f61d390-7b4f-4bc5-a77e-3ea7541f312a" (UID: "5f61d390-7b4f-4bc5-a77e-3ea7541f312a"). InnerVolumeSpecName "kube-api-access-lqls4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:19.018946 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.018925 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-maas-api-xks-ca" pod "154e9e64-a6bd-49f4-8b65-6c6ad5541fe4" (UID: "154e9e64-a6bd-49f4-8b65-6c6ad5541fe4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:19.019032 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.019015 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8f93ca-515a-4785-98e1-1830339ca556-kube-api-access-znc6x" pod "3c8f93ca-515a-4785-98e1-1830339ca556" (UID: "3c8f93ca-515a-4785-98e1-1830339ca556"). InnerVolumeSpecName "kube-api-access-znc6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:19.019076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.019024 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c8f93ca-515a-4785-98e1-1830339ca556-maas-api-xks-ca" pod "3c8f93ca-515a-4785-98e1-1830339ca556" (UID: "3c8f93ca-515a-4785-98e1-1830339ca556"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:19.019295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.019278 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-maas-api-xks-ca" pod "5f61d390-7b4f-4bc5-a77e-3ea7541f312a" (UID: "5f61d390-7b4f-4bc5-a77e-3ea7541f312a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:19.119186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.119157 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:19.119186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.119186 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znc6x\" (UniqueName: \"kubernetes.io/projected/3c8f93ca-515a-4785-98e1-1830339ca556-kube-api-access-znc6x\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:19.119373 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.119196 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:19.119373 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.119208 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqls4\" (UniqueName: \"kubernetes.io/projected/5f61d390-7b4f-4bc5-a77e-3ea7541f312a-kube-api-access-lqls4\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:19.119373 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.119216 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpg9l\" (UniqueName: \"kubernetes.io/projected/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4-kube-api-access-xpg9l\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:19.119373 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.119226 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8f93ca-515a-4785-98e1-1830339ca556-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:19.591554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.591460 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6" Nov 03 20:03:19.591554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.591489 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6" event={"ID":"154e9e64-a6bd-49f4-8b65-6c6ad5541fe4","Type":"ContainerDied","Data":"bbee7156067b53d2a34bbe658106499ddd3793a377324dea46c4634469b8f522"} Nov 03 20:03:19.592080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.591583 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbee7156067b53d2a34bbe658106499ddd3793a377324dea46c4634469b8f522" Nov 03 20:03:19.593022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.593005 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" Nov 03 20:03:19.593105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.593030 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7" event={"ID":"3c8f93ca-515a-4785-98e1-1830339ca556","Type":"ContainerDied","Data":"369c06a341e8d7d963f560b5e7298ac1a7cd99b42e424b184ce72ac9f867484a"} Nov 03 20:03:19.593105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.593062 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="369c06a341e8d7d963f560b5e7298ac1a7cd99b42e424b184ce72ac9f867484a" Nov 03 20:03:19.594519 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.594496 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr" event={"ID":"5f61d390-7b4f-4bc5-a77e-3ea7541f312a","Type":"ContainerDied","Data":"91deed5f18b97472bae04d90db20451d8ce9f86cb2bd46047fb76ace49260534"} Nov 03 20:03:19.594672 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.594557 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91deed5f18b97472bae04d90db20451d8ce9f86cb2bd46047fb76ace49260534" Nov 03 20:03:19.594737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.594709 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr" Nov 03 20:03:19.738125 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.738100 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" Nov 03 20:03:19.828754 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.828727 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2afb37da-7177-4752-93ba-9a68a15a00c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2afb37da-7177-4752-93ba-9a68a15a00c8-maas-api-openshift-ca\") pod \"2afb37da-7177-4752-93ba-9a68a15a00c8\" (UID: \"2afb37da-7177-4752-93ba-9a68a15a00c8\") " Nov 03 20:03:19.828918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.828764 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2afb37da-7177-4752-93ba-9a68a15a00c8-kube-api-access-ff5kq\" (UniqueName: \"kubernetes.io/projected/2afb37da-7177-4752-93ba-9a68a15a00c8-kube-api-access-ff5kq\") pod \"2afb37da-7177-4752-93ba-9a68a15a00c8\" (UID: \"2afb37da-7177-4752-93ba-9a68a15a00c8\") " Nov 03 20:03:19.828918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.828840 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2afb37da-7177-4752-93ba-9a68a15a00c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2afb37da-7177-4752-93ba-9a68a15a00c8-maas-api-xks-ca\") pod \"2afb37da-7177-4752-93ba-9a68a15a00c8\" (UID: \"2afb37da-7177-4752-93ba-9a68a15a00c8\") " Nov 03 20:03:19.829116 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.829091 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2afb37da-7177-4752-93ba-9a68a15a00c8-maas-api-openshift-ca" pod "2afb37da-7177-4752-93ba-9a68a15a00c8" (UID: "2afb37da-7177-4752-93ba-9a68a15a00c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:19.829447 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.829413 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2afb37da-7177-4752-93ba-9a68a15a00c8-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:19.830925 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.830903 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2afb37da-7177-4752-93ba-9a68a15a00c8-maas-api-xks-ca" pod "2afb37da-7177-4752-93ba-9a68a15a00c8" (UID: "2afb37da-7177-4752-93ba-9a68a15a00c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:19.831012 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.830963 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afb37da-7177-4752-93ba-9a68a15a00c8-kube-api-access-ff5kq" pod "2afb37da-7177-4752-93ba-9a68a15a00c8" (UID: "2afb37da-7177-4752-93ba-9a68a15a00c8"). InnerVolumeSpecName "kube-api-access-ff5kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:19.930711 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.930679 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff5kq\" (UniqueName: \"kubernetes.io/projected/2afb37da-7177-4752-93ba-9a68a15a00c8-kube-api-access-ff5kq\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:19.930711 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:19.930708 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2afb37da-7177-4752-93ba-9a68a15a00c8-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:20.603193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:20.603110 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" Nov 03 20:03:20.603193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:20.603147 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4" event={"ID":"2afb37da-7177-4752-93ba-9a68a15a00c8","Type":"ContainerDied","Data":"badb0b270e75d8baa881b3b22bb6220562e160e323cb984db300ddafffebc40e"} Nov 03 20:03:20.603707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:20.603197 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="badb0b270e75d8baa881b3b22bb6220562e160e323cb984db300ddafffebc40e" Nov 03 20:03:22.070415 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.070381 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch"] Nov 03 20:03:22.070990 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.070975 5111 cpu_manager.go:414] "containerMap: removing container" podUID="5f61d390-7b4f-4bc5-a77e-3ea7541f312a" containerName="revoke-keys" Nov 03 20:03:22.071050 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.070991 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f61d390-7b4f-4bc5-a77e-3ea7541f312a" containerName="revoke-keys" Nov 03 20:03:22.071050 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.071020 5111 cpu_manager.go:414] "containerMap: removing container" podUID="3c8f93ca-515a-4785-98e1-1830339ca556" containerName="revoke-keys" Nov 03 20:03:22.071050 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.071029 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8f93ca-515a-4785-98e1-1830339ca556" containerName="revoke-keys" Nov 03 20:03:22.071050 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.071040 5111 cpu_manager.go:414] "containerMap: removing container" podUID="154e9e64-a6bd-49f4-8b65-6c6ad5541fe4" containerName="revoke-keys" Nov 03 20:03:22.071050 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.071045 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154e9e64-a6bd-49f4-8b65-6c6ad5541fe4" containerName="revoke-keys" Nov 03 20:03:22.071188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.071056 5111 cpu_manager.go:414] "containerMap: removing container" podUID="2afb37da-7177-4752-93ba-9a68a15a00c8" containerName="revoke-keys" Nov 03 20:03:22.071188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.071061 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2afb37da-7177-4752-93ba-9a68a15a00c8" containerName="revoke-keys" Nov 03 20:03:22.071188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.071158 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8f93ca-515a-4785-98e1-1830339ca556" containerName="revoke-keys" Nov 03 20:03:22.071188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.071165 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="154e9e64-a6bd-49f4-8b65-6c6ad5541fe4" containerName="revoke-keys" Nov 03 20:03:22.071188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.071172 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="2afb37da-7177-4752-93ba-9a68a15a00c8" containerName="revoke-keys" Nov 03 20:03:22.071188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.071184 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f61d390-7b4f-4bc5-a77e-3ea7541f312a" containerName="revoke-keys" Nov 03 20:03:22.074650 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.074622 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.091653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.091629 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch"] Nov 03 20:03:22.157795 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.157759 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330ae291-dce5-435f-aac6-5cbc4214c273-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.157944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.157826 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330ae291-dce5-435f-aac6-5cbc4214c273-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.157944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.157923 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.158036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.157955 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330ae291-dce5-435f-aac6-5cbc4214c273-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.158071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.158049 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.158105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.158077 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zdfl\" (UniqueName: \"kubernetes.io/projected/330ae291-dce5-435f-aac6-5cbc4214c273-kube-api-access-2zdfl\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.158167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.158154 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.158205 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.158184 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.158263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.158250 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.252192 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.252155 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk"] Nov 03 20:03:22.257348 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.257321 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.259405 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.259375 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zdfl\" (UniqueName: \"kubernetes.io/projected/330ae291-dce5-435f-aac6-5cbc4214c273-kube-api-access-2zdfl\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.259556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.259490 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.259556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.259549 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.259684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.259616 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.259750 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.259717 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330ae291-dce5-435f-aac6-5cbc4214c273-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.259827 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.259811 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330ae291-dce5-435f-aac6-5cbc4214c273-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.259878 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.259866 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.259927 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.259895 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330ae291-dce5-435f-aac6-5cbc4214c273-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.259999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.259978 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.260048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.260006 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.260369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.260347 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.261051 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.261023 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.261158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.261142 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.261563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.261510 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330ae291-dce5-435f-aac6-5cbc4214c273-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.262959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.262940 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330ae291-dce5-435f-aac6-5cbc4214c273-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.263521 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.263500 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.273588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.273563 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk"] Nov 03 20:03:22.283095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.283048 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zdfl\" (UniqueName: \"kubernetes.io/projected/330ae291-dce5-435f-aac6-5cbc4214c273-kube-api-access-2zdfl\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.286300 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.286261 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330ae291-dce5-435f-aac6-5cbc4214c273-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.361750 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.361661 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.361750 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.361698 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.361750 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.361716 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.362009 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.361777 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.362009 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.361895 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smfr2\" (UniqueName: \"kubernetes.io/projected/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-kube-api-access-smfr2\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.362103 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.362025 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.362103 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.362081 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.362214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.362199 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.362255 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.362244 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.402935 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.402911 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:22.469653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.463451 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.469653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.463482 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.469653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.463508 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.469653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.463591 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smfr2\" (UniqueName: \"kubernetes.io/projected/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-kube-api-access-smfr2\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.469653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.463691 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.469653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.463738 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.469653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.463864 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.469653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.463908 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.469653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.464009 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.469653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.465312 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.472205 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.472175 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.485209 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.485179 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.485356 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.485337 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.485460 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.485441 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.485580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.485560 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.489001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.488936 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.489099 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.488992 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.489344 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.489319 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smfr2\" (UniqueName: \"kubernetes.io/projected/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-kube-api-access-smfr2\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.627023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.626991 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:22.693496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.693269 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch"] Nov 03 20:03:22.695023 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:03:22.693827 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330ae291_dce5_435f_aac6_5cbc4214c273.slice/crio-a3c54643ca7fb49071443e186458da38fd4d77253fccd1de4f56a5ef5f816d1d WatchSource:0}: Error finding container a3c54643ca7fb49071443e186458da38fd4d77253fccd1de4f56a5ef5f816d1d: Status 404 returned error can't find the container with id a3c54643ca7fb49071443e186458da38fd4d77253fccd1de4f56a5ef5f816d1d Nov 03 20:03:22.698629 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.697784 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:03:22.698629 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.697872 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:03:22.698629 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.697914 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:03:22.908968 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.908871 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk"] Nov 03 20:03:22.914049 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:03:22.914013 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a90be1_f894_4f95_8ccc_f8d2bd2d929d.slice/crio-7ee22f14e131102f3774ea2271556cde9479a0f8617e3aacbbbb33ea8c51aaeb WatchSource:0}: Error finding container 7ee22f14e131102f3774ea2271556cde9479a0f8617e3aacbbbb33ea8c51aaeb: Status 404 returned error can't find the container with id 7ee22f14e131102f3774ea2271556cde9479a0f8617e3aacbbbb33ea8c51aaeb Nov 03 20:03:22.917432 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.917387 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:03:22.917520 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.917486 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:03:22.917574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:22.917550 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:03:23.659744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:23.659698 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" event={"ID":"48a90be1-f894-4f95-8ccc-f8d2bd2d929d","Type":"ContainerStarted","Data":"3da3f0cbd0993adee23c0349751ff21b3ea92ed64537be82304bfeb73f7be019"} Nov 03 20:03:23.660398 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:23.660371 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" event={"ID":"48a90be1-f894-4f95-8ccc-f8d2bd2d929d","Type":"ContainerStarted","Data":"7ee22f14e131102f3774ea2271556cde9479a0f8617e3aacbbbb33ea8c51aaeb"} Nov 03 20:03:23.695552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:23.694667 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" event={"ID":"330ae291-dce5-435f-aac6-5cbc4214c273","Type":"ContainerStarted","Data":"d9aa98982d2c7b6b3083db68240b1f078c8e6492075c0648a8fb82236fbf31db"} Nov 03 20:03:23.695552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:23.694742 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" event={"ID":"330ae291-dce5-435f-aac6-5cbc4214c273","Type":"ContainerStarted","Data":"a3c54643ca7fb49071443e186458da38fd4d77253fccd1de4f56a5ef5f816d1d"} Nov 03 20:03:23.709564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:23.706452 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" podStartSLOduration=1.706434098 podStartE2EDuration="1.706434098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:23.704095793 +0000 UTC m=+1433.666474233" watchObservedRunningTime="2026-11-03 20:03:23.706434098 +0000 UTC m=+1433.668812540" Nov 03 20:03:23.731332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:23.731278 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" podStartSLOduration=1.731264641 podStartE2EDuration="1.731264641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:23.728438094 +0000 UTC m=+1433.690816534" watchObservedRunningTime="2026-11-03 20:03:23.731264641 +0000 UTC m=+1433.693643079" Nov 03 20:03:24.414555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:24.414276 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:24.414555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:24.414415 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:03:24.414555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:24.414457 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" podUID="330ae291-dce5-435f-aac6-5cbc4214c273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:03:24.668803 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:24.668706 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:03:24.669355 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:24.669320 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" podUID="48a90be1-f894-4f95-8ccc-f8d2bd2d929d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 20:03:24.674610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:24.670367 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:25.400556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:25.379506 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:25.400556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:25.379892 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:25.425558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:25.407058 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:03:25.425558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:25.407117 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" podUID="330ae291-dce5-435f-aac6-5cbc4214c273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:03:25.475556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:25.442800 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:25.475556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:25.473845 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:25.475556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:25.474243 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded" Nov 03 20:03:25.520557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:25.517739 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:25.520557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:25.519071 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:03:25.648557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:25.634973 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:03:25.648557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:25.635038 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" podUID="48a90be1-f894-4f95-8ccc-f8d2bd2d929d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 20:03:25.681615 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:25.680785 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:25.706560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:25.702161 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:25.769557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:25.768680 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:03:26.391556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.382685 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:26.407714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.407677 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:03:26.407983 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.407952 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" podUID="330ae291-dce5-435f-aac6-5cbc4214c273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:03:26.476791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.461621 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:26.478692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.478642 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:26.526565 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.521779 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:26.526565 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.522010 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 20:03:26.628581 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.628494 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:03:26.628943 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.628889 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" podUID="48a90be1-f894-4f95-8ccc-f8d2bd2d929d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 20:03:26.710625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.709480 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:26.733703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.733673 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:03:26.733914 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.733901 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr"] Nov 03 20:03:26.817569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.816072 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:03:26.817569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.816121 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr"] Nov 03 20:03:26.817569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.816177 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:03:26.817569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.816213 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:03:26.817569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.816248 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:03:26.817569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.816265 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27"] Nov 03 20:03:26.817569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.816866 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr" Nov 03 20:03:26.825318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.825291 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:03:26.826189 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.826168 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:03:26.826312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.826203 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27"] Nov 03 20:03:26.826312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.826232 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:03:26.826312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.826247 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:03:26.826312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.826292 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:03:26.826312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.826308 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:03:26.826585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.826328 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:03:26.826585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.826402 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27" Nov 03 20:03:26.828197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.828175 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:03:26.828349 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.828267 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:03:26.828446 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.828394 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:03:26.828609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.828581 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:03:26.841742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.841713 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edca7308-8c50-4292-9b4d-43dbb4334b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7578f3b-pzxdr\" (UID: \"edca7308-8c50-4292-9b4d-43dbb4334b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr" Nov 03 20:03:26.841990 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.841970 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edca7308-8c50-4292-9b4d-43dbb4334b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7578f3b-pzxdr\" (UID: \"edca7308-8c50-4292-9b4d-43dbb4334b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr" Nov 03 20:03:26.842232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.842215 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r72mb\" (UniqueName: \"kubernetes.io/projected/edca7308-8c50-4292-9b4d-43dbb4334b55-kube-api-access-r72mb\") pod \"maas-api-revoke-sub-f7578f3b-pzxdr\" (UID: \"edca7308-8c50-4292-9b4d-43dbb4334b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr" Nov 03 20:03:27.015610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.991879 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f790e958-814f-41c6-8675-71de6221a2cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e00cce-nnv27\" (UID: \"f790e958-814f-41c6-8675-71de6221a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27" Nov 03 20:03:27.015610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.992011 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edca7308-8c50-4292-9b4d-43dbb4334b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7578f3b-pzxdr\" (UID: \"edca7308-8c50-4292-9b4d-43dbb4334b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr" Nov 03 20:03:27.015610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.992088 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edca7308-8c50-4292-9b4d-43dbb4334b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7578f3b-pzxdr\" (UID: \"edca7308-8c50-4292-9b4d-43dbb4334b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr" Nov 03 20:03:27.015610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.992130 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2t6k\" (UniqueName: \"kubernetes.io/projected/f790e958-814f-41c6-8675-71de6221a2cc-kube-api-access-r2t6k\") pod \"maas-api-revoke-sub-b3e00cce-nnv27\" (UID: \"f790e958-814f-41c6-8675-71de6221a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27" Nov 03 20:03:27.015610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.992415 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r72mb\" (UniqueName: \"kubernetes.io/projected/edca7308-8c50-4292-9b4d-43dbb4334b55-kube-api-access-r72mb\") pod \"maas-api-revoke-sub-f7578f3b-pzxdr\" (UID: \"edca7308-8c50-4292-9b4d-43dbb4334b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr" Nov 03 20:03:27.015610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:26.992587 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f790e958-814f-41c6-8675-71de6221a2cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e00cce-nnv27\" (UID: \"f790e958-814f-41c6-8675-71de6221a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27" Nov 03 20:03:27.015610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.005322 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edca7308-8c50-4292-9b4d-43dbb4334b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7578f3b-pzxdr\" (UID: \"edca7308-8c50-4292-9b4d-43dbb4334b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr" Nov 03 20:03:27.020314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.020276 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r72mb\" (UniqueName: \"kubernetes.io/projected/edca7308-8c50-4292-9b4d-43dbb4334b55-kube-api-access-r72mb\") pod \"maas-api-revoke-sub-f7578f3b-pzxdr\" (UID: \"edca7308-8c50-4292-9b4d-43dbb4334b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr" Nov 03 20:03:27.026040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.026009 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edca7308-8c50-4292-9b4d-43dbb4334b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7578f3b-pzxdr\" (UID: \"edca7308-8c50-4292-9b4d-43dbb4334b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr" Nov 03 20:03:27.094357 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.094311 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f790e958-814f-41c6-8675-71de6221a2cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e00cce-nnv27\" (UID: \"f790e958-814f-41c6-8675-71de6221a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27" Nov 03 20:03:27.097824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.097792 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f790e958-814f-41c6-8675-71de6221a2cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e00cce-nnv27\" (UID: \"f790e958-814f-41c6-8675-71de6221a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27" Nov 03 20:03:27.098084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.098069 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2t6k\" (UniqueName: \"kubernetes.io/projected/f790e958-814f-41c6-8675-71de6221a2cc-kube-api-access-r2t6k\") pod \"maas-api-revoke-sub-b3e00cce-nnv27\" (UID: \"f790e958-814f-41c6-8675-71de6221a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27" Nov 03 20:03:27.111501 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.099341 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f790e958-814f-41c6-8675-71de6221a2cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e00cce-nnv27\" (UID: \"f790e958-814f-41c6-8675-71de6221a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27" Nov 03 20:03:27.112574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.112284 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f790e958-814f-41c6-8675-71de6221a2cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e00cce-nnv27\" (UID: \"f790e958-814f-41c6-8675-71de6221a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27" Nov 03 20:03:27.116254 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.116223 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2t6k\" (UniqueName: \"kubernetes.io/projected/f790e958-814f-41c6-8675-71de6221a2cc-kube-api-access-r2t6k\") pod \"maas-api-revoke-sub-b3e00cce-nnv27\" (UID: \"f790e958-814f-41c6-8675-71de6221a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27" Nov 03 20:03:27.144344 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.144309 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr" Nov 03 20:03:27.216273 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.216176 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27" Nov 03 20:03:27.405956 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.405918 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:03:27.406142 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.406010 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" podUID="330ae291-dce5-435f-aac6-5cbc4214c273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:03:27.544342 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.544315 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr"] Nov 03 20:03:27.580626 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:03:27.577667 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedca7308_8c50_4292_9b4d_43dbb4334b55.slice/crio-308dc3bf556889838563f911e76fa4a1b97470121a4c7896237bf0793594e741 WatchSource:0}: Error finding container 308dc3bf556889838563f911e76fa4a1b97470121a4c7896237bf0793594e741: Status 404 returned error can't find the container with id 308dc3bf556889838563f911e76fa4a1b97470121a4c7896237bf0793594e741 Nov 03 20:03:27.615305 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.615005 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27"] Nov 03 20:03:27.628471 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.628442 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:03:27.628602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.628497 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" podUID="48a90be1-f894-4f95-8ccc-f8d2bd2d929d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 20:03:27.923600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.923560 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27" event={"ID":"f790e958-814f-41c6-8675-71de6221a2cc","Type":"ContainerStarted","Data":"780b221698a697cb0764cbf7b8d27adbeb0dc14dc6dc66d02f23590c8dcf8739"} Nov 03 20:03:27.924773 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:27.924750 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr" event={"ID":"edca7308-8c50-4292-9b4d-43dbb4334b55","Type":"ContainerStarted","Data":"308dc3bf556889838563f911e76fa4a1b97470121a4c7896237bf0793594e741"} Nov 03 20:03:28.404260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:28.404057 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:03:28.404260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:28.404128 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" podUID="330ae291-dce5-435f-aac6-5cbc4214c273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:03:28.628077 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:28.627745 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:03:28.628077 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:28.627794 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" podUID="48a90be1-f894-4f95-8ccc-f8d2bd2d929d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 20:03:28.933199 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:28.933161 5111 generic.go:356] "Generic (PLEG): container finished" podID="f790e958-814f-41c6-8675-71de6221a2cc" containerID="6c5b084cf045679bcd8c9bb4280b8b951408ddd6f94e3c62c5550d380f1c7c53" exitCode=0 Nov 03 20:03:28.933636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:28.933242 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27" event={"ID":"f790e958-814f-41c6-8675-71de6221a2cc","Type":"ContainerDied","Data":"6c5b084cf045679bcd8c9bb4280b8b951408ddd6f94e3c62c5550d380f1c7c53"} Nov 03 20:03:28.934811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:28.934790 5111 generic.go:356] "Generic (PLEG): container finished" podID="edca7308-8c50-4292-9b4d-43dbb4334b55" containerID="e2b41cf3186dfd4e278fc366a37e5bc83ff9387d52435ff1827a542ad381804e" exitCode=0 Nov 03 20:03:28.934871 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:28.934821 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr" event={"ID":"edca7308-8c50-4292-9b4d-43dbb4334b55","Type":"ContainerDied","Data":"e2b41cf3186dfd4e278fc366a37e5bc83ff9387d52435ff1827a542ad381804e"} Nov 03 20:03:29.404087 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:29.404046 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:03:29.404245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:29.404108 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" podUID="330ae291-dce5-435f-aac6-5cbc4214c273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:03:29.628166 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:29.628126 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:03:29.628325 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:29.628181 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" podUID="48a90be1-f894-4f95-8ccc-f8d2bd2d929d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 20:03:30.140592 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.140418 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr" Nov 03 20:03:30.147543 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.146713 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edca7308-8c50-4292-9b4d-43dbb4334b55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edca7308-8c50-4292-9b4d-43dbb4334b55-maas-api-openshift-ca\") pod \"edca7308-8c50-4292-9b4d-43dbb4334b55\" (UID: \"edca7308-8c50-4292-9b4d-43dbb4334b55\") " Nov 03 20:03:30.147543 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.146787 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edca7308-8c50-4292-9b4d-43dbb4334b55-kube-api-access-r72mb\" (UniqueName: \"kubernetes.io/projected/edca7308-8c50-4292-9b4d-43dbb4334b55-kube-api-access-r72mb\") pod \"edca7308-8c50-4292-9b4d-43dbb4334b55\" (UID: \"edca7308-8c50-4292-9b4d-43dbb4334b55\") " Nov 03 20:03:30.147543 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.146823 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edca7308-8c50-4292-9b4d-43dbb4334b55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edca7308-8c50-4292-9b4d-43dbb4334b55-maas-api-xks-ca\") pod \"edca7308-8c50-4292-9b4d-43dbb4334b55\" (UID: \"edca7308-8c50-4292-9b4d-43dbb4334b55\") " Nov 03 20:03:30.148785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.148488 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edca7308-8c50-4292-9b4d-43dbb4334b55-maas-api-openshift-ca" pod "edca7308-8c50-4292-9b4d-43dbb4334b55" (UID: "edca7308-8c50-4292-9b4d-43dbb4334b55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:30.149770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.149743 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edca7308-8c50-4292-9b4d-43dbb4334b55-maas-api-xks-ca" pod "edca7308-8c50-4292-9b4d-43dbb4334b55" (UID: "edca7308-8c50-4292-9b4d-43dbb4334b55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:30.154468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.154432 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edca7308-8c50-4292-9b4d-43dbb4334b55-kube-api-access-r72mb" pod "edca7308-8c50-4292-9b4d-43dbb4334b55" (UID: "edca7308-8c50-4292-9b4d-43dbb4334b55"). InnerVolumeSpecName "kube-api-access-r72mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:30.203059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.202805 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27" Nov 03 20:03:30.248374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.248336 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f790e958-814f-41c6-8675-71de6221a2cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f790e958-814f-41c6-8675-71de6221a2cc-maas-api-openshift-ca\") pod \"f790e958-814f-41c6-8675-71de6221a2cc\" (UID: \"f790e958-814f-41c6-8675-71de6221a2cc\") " Nov 03 20:03:30.248606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.248507 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f790e958-814f-41c6-8675-71de6221a2cc-kube-api-access-r2t6k\" (UniqueName: \"kubernetes.io/projected/f790e958-814f-41c6-8675-71de6221a2cc-kube-api-access-r2t6k\") pod \"f790e958-814f-41c6-8675-71de6221a2cc\" (UID: \"f790e958-814f-41c6-8675-71de6221a2cc\") " Nov 03 20:03:30.248606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.248558 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f790e958-814f-41c6-8675-71de6221a2cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f790e958-814f-41c6-8675-71de6221a2cc-maas-api-xks-ca\") pod \"f790e958-814f-41c6-8675-71de6221a2cc\" (UID: \"f790e958-814f-41c6-8675-71de6221a2cc\") " Nov 03 20:03:30.248788 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.248763 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f790e958-814f-41c6-8675-71de6221a2cc-maas-api-openshift-ca" pod "f790e958-814f-41c6-8675-71de6221a2cc" (UID: "f790e958-814f-41c6-8675-71de6221a2cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:30.249042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.249026 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edca7308-8c50-4292-9b4d-43dbb4334b55-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:30.249110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.249046 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r72mb\" (UniqueName: \"kubernetes.io/projected/edca7308-8c50-4292-9b4d-43dbb4334b55-kube-api-access-r72mb\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:30.249110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.249060 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edca7308-8c50-4292-9b4d-43dbb4334b55-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:30.249110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.249075 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f790e958-814f-41c6-8675-71de6221a2cc-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:30.253860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.253830 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f790e958-814f-41c6-8675-71de6221a2cc-maas-api-xks-ca" pod "f790e958-814f-41c6-8675-71de6221a2cc" (UID: "f790e958-814f-41c6-8675-71de6221a2cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:30.253992 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.253960 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f790e958-814f-41c6-8675-71de6221a2cc-kube-api-access-r2t6k" pod "f790e958-814f-41c6-8675-71de6221a2cc" (UID: "f790e958-814f-41c6-8675-71de6221a2cc"). InnerVolumeSpecName "kube-api-access-r2t6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:30.354652 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.354614 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2t6k\" (UniqueName: \"kubernetes.io/projected/f790e958-814f-41c6-8675-71de6221a2cc-kube-api-access-r2t6k\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:30.354652 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.354653 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f790e958-814f-41c6-8675-71de6221a2cc-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:30.404329 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.404293 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:03:30.404493 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.404356 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" podUID="330ae291-dce5-435f-aac6-5cbc4214c273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:03:30.961487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.961451 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27" Nov 03 20:03:30.961487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.961468 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27" event={"ID":"f790e958-814f-41c6-8675-71de6221a2cc","Type":"ContainerDied","Data":"780b221698a697cb0764cbf7b8d27adbeb0dc14dc6dc66d02f23590c8dcf8739"} Nov 03 20:03:30.961755 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.961527 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="780b221698a697cb0764cbf7b8d27adbeb0dc14dc6dc66d02f23590c8dcf8739" Nov 03 20:03:30.972706 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.971968 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr" event={"ID":"edca7308-8c50-4292-9b4d-43dbb4334b55","Type":"ContainerDied","Data":"308dc3bf556889838563f911e76fa4a1b97470121a4c7896237bf0793594e741"} Nov 03 20:03:30.972706 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.972052 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="308dc3bf556889838563f911e76fa4a1b97470121a4c7896237bf0793594e741" Nov 03 20:03:30.972706 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:30.972125 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr" Nov 03 20:03:31.398661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.398250 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz"] Nov 03 20:03:31.417496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.399914 5111 cpu_manager.go:414] "containerMap: removing container" podUID="edca7308-8c50-4292-9b4d-43dbb4334b55" containerName="revoke-keys" Nov 03 20:03:31.417496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.399942 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edca7308-8c50-4292-9b4d-43dbb4334b55" containerName="revoke-keys" Nov 03 20:03:31.417496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.399989 5111 cpu_manager.go:414] "containerMap: removing container" podUID="f790e958-814f-41c6-8675-71de6221a2cc" containerName="revoke-keys" Nov 03 20:03:31.417496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.399998 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f790e958-814f-41c6-8675-71de6221a2cc" containerName="revoke-keys" Nov 03 20:03:31.417496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.400148 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="edca7308-8c50-4292-9b4d-43dbb4334b55" containerName="revoke-keys" Nov 03 20:03:31.417496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.400169 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="f790e958-814f-41c6-8675-71de6221a2cc" containerName="revoke-keys" Nov 03 20:03:31.424303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.419785 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.426176 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.426146 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-120fe3\"/\"kube-root-ca.crt\"" Nov 03 20:03:31.426304 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.426271 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-120fe3\"/\"openshift-service-ca.crt\"" Nov 03 20:03:31.426367 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.426339 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-120fe3\"/\"distinct-w0-120fe3-kserve-self-signed-certs\"" Nov 03 20:03:31.426434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.426380 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-120fe3\"/\"default-dockercfg-c5bvs\"" Nov 03 20:03:31.427822 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.427792 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz"] Nov 03 20:03:31.644557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.644126 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b53a56a-77d0-4fbe-b767-82563feb20a0-tls-certs\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.644557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.644207 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-model-cache\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.644557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.644367 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-dshm\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.644557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.644427 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vdd4\" (UniqueName: \"kubernetes.io/projected/8b53a56a-77d0-4fbe-b767-82563feb20a0-kube-api-access-5vdd4\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.644941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.644632 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-home\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.644941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.644686 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-tmp-dir\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.694623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.692567 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:03:31.694623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.692649 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" podUID="48a90be1-f894-4f95-8ccc-f8d2bd2d929d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:31.719966 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.719913 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:31.721060 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.721033 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:31.722798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.722781 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:03:31.748368 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.748324 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b53a56a-77d0-4fbe-b767-82563feb20a0-tls-certs\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.749839 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.749795 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-model-cache\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.749969 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.749917 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-dshm\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.750031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.749974 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vdd4\" (UniqueName: \"kubernetes.io/projected/8b53a56a-77d0-4fbe-b767-82563feb20a0-kube-api-access-5vdd4\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.750172 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.750154 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-home\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.750265 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.750203 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-tmp-dir\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.750742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.750719 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-tmp-dir\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.751014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.750994 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-model-cache\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.762108 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.761605 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b53a56a-77d0-4fbe-b767-82563feb20a0-tls-certs\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.762253 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.762148 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-home\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.767846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.767815 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-dshm\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.809576 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.792015 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vdd4\" (UniqueName: \"kubernetes.io/projected/8b53a56a-77d0-4fbe-b767-82563feb20a0-kube-api-access-5vdd4\") pod \"distinct-w0-120fe3-kserve-86b546d956-5lxfz\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:31.871623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:31.871329 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:32.101171 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:32.101140 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:32.101600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:32.101579 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:32.102945 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:32.102920 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:03:32.174339 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:32.174191 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz"] Nov 03 20:03:32.991895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:32.991851 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" event={"ID":"8b53a56a-77d0-4fbe-b767-82563feb20a0","Type":"ContainerStarted","Data":"156598ce7679cd5481d4713a9a82a72eb4fde02a1e5f9cf6f546972c8af72fc4"} Nov 03 20:03:32.991895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:32.991900 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" event={"ID":"8b53a56a-77d0-4fbe-b767-82563feb20a0","Type":"ContainerStarted","Data":"2d1140584a063fbb73f815afdff9d8bea53869ed22628e52a70187dedf5d4d2c"} Nov 03 20:03:33.009646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:33.009590 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podStartSLOduration=2.009576859 podStartE2EDuration="2.009576859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:33.007327645 +0000 UTC m=+1442.969706098" watchObservedRunningTime="2026-11-03 20:03:33.009576859 +0000 UTC m=+1442.971955298" Nov 03 20:03:33.872462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:33.872425 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:35.057337 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:03:35.057290 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:03:37.377488 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.377458 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss"] Nov 03 20:03:37.386758 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.386728 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.413966 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.403082 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss"] Nov 03 20:03:37.442317 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.442271 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.442518 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.442347 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.442518 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.442376 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.442518 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.442466 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.442714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.442517 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.442714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.442673 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.442805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.442740 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.442805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.442767 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqtj7\" (UniqueName: \"kubernetes.io/projected/dd161e75-9c5f-4e1c-8a76-5a0171162c42-kube-api-access-nqtj7\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.442894 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.442834 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.544184 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.544144 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.544378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.544191 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.544378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.544214 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.544378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.544288 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.544378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.544333 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.544689 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.544448 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.544689 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.544511 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.544689 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.544577 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqtj7\" (UniqueName: \"kubernetes.io/projected/dd161e75-9c5f-4e1c-8a76-5a0171162c42-kube-api-access-nqtj7\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.544689 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.544580 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.544949 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.544750 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.544949 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.544937 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.545042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.544974 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.545138 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.545113 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.545314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.545296 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.546778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.546757 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.547174 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.547156 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.552248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.552222 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.552338 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.552276 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqtj7\" (UniqueName: \"kubernetes.io/projected/dd161e75-9c5f-4e1c-8a76-5a0171162c42-kube-api-access-nqtj7\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:37.718275 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:37.718195 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:38.012283 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:38.012254 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss"] Nov 03 20:03:38.012422 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:03:38.012360 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd161e75_9c5f_4e1c_8a76_5a0171162c42.slice/crio-e63f30249eb4b1d0e72fda445a0379f40da5231480a5a6721dd5e52da118d424 WatchSource:0}: Error finding container e63f30249eb4b1d0e72fda445a0379f40da5231480a5a6721dd5e52da118d424: Status 404 returned error can't find the container with id e63f30249eb4b1d0e72fda445a0379f40da5231480a5a6721dd5e52da118d424 Nov 03 20:03:38.015149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:38.015112 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:03:38.015279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:38.015198 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:03:38.015279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:38.015240 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:03:38.036782 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:38.036751 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" event={"ID":"dd161e75-9c5f-4e1c-8a76-5a0171162c42","Type":"ContainerStarted","Data":"e63f30249eb4b1d0e72fda445a0379f40da5231480a5a6721dd5e52da118d424"} Nov 03 20:03:39.083641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:39.082455 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" event={"ID":"dd161e75-9c5f-4e1c-8a76-5a0171162c42","Type":"ContainerStarted","Data":"be379da620d03d13c5d4c536bd4776b855f69a63874df9b83a6ce0c0066b5a82"} Nov 03 20:03:39.107576 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:39.107495 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" podStartSLOduration=2.107473592 podStartE2EDuration="2.107473592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:39.103328071 +0000 UTC m=+1449.065706510" watchObservedRunningTime="2026-11-03 20:03:39.107473592 +0000 UTC m=+1449.069852051" Nov 03 20:03:39.718780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:39.718744 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:39.720736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:39.720709 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:03:39.720883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:39.720766 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" podUID="dd161e75-9c5f-4e1c-8a76-5a0171162c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 20:03:40.719048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:40.719010 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:03:40.719464 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:40.719064 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" podUID="dd161e75-9c5f-4e1c-8a76-5a0171162c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 20:03:41.722983 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:41.721054 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:03:41.722983 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:41.721122 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" podUID="dd161e75-9c5f-4e1c-8a76-5a0171162c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 20:03:41.874557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:41.874162 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:42.718955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:42.718916 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:03:42.719262 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:42.719230 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" podUID="dd161e75-9c5f-4e1c-8a76-5a0171162c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 20:03:42.794513 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:42.793976 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:42.817349 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:42.817288 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65"] Nov 03 20:03:42.946237 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:42.881637 5111 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": context deadline exceeded" Nov 03 20:03:42.997633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:42.981782 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.009628 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.006750 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-11a5b0\"/\"distinct-w2-11a5b0-kserve-self-signed-certs\"" Nov 03 20:03:43.221181 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.221139 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-tmp-dir\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.221487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.221464 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4b3178-37d9-4185-ab51-19e7e7914725-tls-certs\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.221724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.221704 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb5nl\" (UniqueName: \"kubernetes.io/projected/4c4b3178-37d9-4185-ab51-19e7e7914725-kube-api-access-gb5nl\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.221864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.221849 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-dshm\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.221985 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.221971 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-model-cache\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.229740 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.229707 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-home\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.296639 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.290152 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65"] Nov 03 20:03:43.354556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.354300 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4b3178-37d9-4185-ab51-19e7e7914725-tls-certs\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.354556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.354459 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb5nl\" (UniqueName: \"kubernetes.io/projected/4c4b3178-37d9-4185-ab51-19e7e7914725-kube-api-access-gb5nl\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.354556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.354521 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-dshm\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.354901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.354613 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-model-cache\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.354901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.354683 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-home\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.354901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.354747 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-tmp-dir\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.362557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.355242 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-tmp-dir\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.362557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.356431 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-model-cache\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.362557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.356722 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-home\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.383798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.381817 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-dshm\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.384827 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.384794 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb5nl\" (UniqueName: \"kubernetes.io/projected/4c4b3178-37d9-4185-ab51-19e7e7914725-kube-api-access-gb5nl\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.425563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.402630 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4b3178-37d9-4185-ab51-19e7e7914725-tls-certs\") pod \"distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.445556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.439652 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:03:43.449555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.447764 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:43.476555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.475614 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:43.532312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.530722 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:43.629557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.628774 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:43.679386 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.679342 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:43.737736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.729885 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:03:43.737736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.729947 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" podUID="dd161e75-9c5f-4e1c-8a76-5a0171162c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 20:03:43.796619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.796481 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:43.797131 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.797066 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:03:43.842479 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.842451 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:03:43.897048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:43.897009 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:44.185575 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:03:44.185503 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c4b3178_37d9_4185_ab51_19e7e7914725.slice/crio-02f876c17622aca0cc146f2b120aa7df72519106008d5649cb0a59548ec85ba6 WatchSource:0}: Error finding container 02f876c17622aca0cc146f2b120aa7df72519106008d5649cb0a59548ec85ba6: Status 404 returned error can't find the container with id 02f876c17622aca0cc146f2b120aa7df72519106008d5649cb0a59548ec85ba6 Nov 03 20:03:44.188309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:44.188284 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65"] Nov 03 20:03:44.304107 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:44.304064 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" event={"ID":"4c4b3178-37d9-4185-ab51-19e7e7914725","Type":"ContainerStarted","Data":"9323b22098eb0ae96f33e839bfe39ccb7b93b0b96508e029e2f612b91bec95cc"} Nov 03 20:03:44.304245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:44.304115 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" event={"ID":"4c4b3178-37d9-4185-ab51-19e7e7914725","Type":"ContainerStarted","Data":"02f876c17622aca0cc146f2b120aa7df72519106008d5649cb0a59548ec85ba6"} Nov 03 20:03:44.323033 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:44.322975 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podStartSLOduration=2.322957348 podStartE2EDuration="2.322957348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:44.321300336 +0000 UTC m=+1454.283678778" watchObservedRunningTime="2026-11-03 20:03:44.322957348 +0000 UTC m=+1454.285335786" Nov 03 20:03:44.630084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:44.630057 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:44.718626 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:44.718591 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:03:44.718802 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:44.718652 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" podUID="dd161e75-9c5f-4e1c-8a76-5a0171162c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 20:03:45.689154 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.682756 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd"] Nov 03 20:03:45.691326 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.691053 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.691843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.691816 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd"] Nov 03 20:03:45.696474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.696449 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5348ca\"/\"distinct-w4-5348ca-kserve-self-signed-certs\"" Nov 03 20:03:45.697468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.697104 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5348ca\"/\"kube-root-ca.crt\"" Nov 03 20:03:45.697468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.697156 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5348ca\"/\"openshift-service-ca.crt\"" Nov 03 20:03:45.697468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.697198 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5348ca\"/\"default-dockercfg-2h9k8\"" Nov 03 20:03:45.821627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.821590 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f264669-2468-4f77-b3a4-0eea66f35352-tls-certs\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.821822 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.821736 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkg82\" (UniqueName: \"kubernetes.io/projected/8f264669-2468-4f77-b3a4-0eea66f35352-kube-api-access-kkg82\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.821822 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.821818 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-dshm\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.821939 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.821904 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-model-cache\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.821992 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.821970 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-tmp-dir\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.822038 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.822027 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-home\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.924071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.924026 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-model-cache\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.924278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.924106 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-tmp-dir\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.924278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.924174 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-home\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.924278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.924255 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f264669-2468-4f77-b3a4-0eea66f35352-tls-certs\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.924434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.924389 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkg82\" (UniqueName: \"kubernetes.io/projected/8f264669-2468-4f77-b3a4-0eea66f35352-kube-api-access-kkg82\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.924510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.924486 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-dshm\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.924724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.924667 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-model-cache\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.924724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.924705 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-tmp-dir\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.924900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.924741 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-home\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.926984 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.926960 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-dshm\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.927241 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.927225 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f264669-2468-4f77-b3a4-0eea66f35352-tls-certs\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:45.932756 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:45.932733 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkg82\" (UniqueName: \"kubernetes.io/projected/8f264669-2468-4f77-b3a4-0eea66f35352-kube-api-access-kkg82\") pod \"distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:46.008034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:46.007929 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:46.322933 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:46.322907 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd"] Nov 03 20:03:46.324721 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:03:46.324679 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f264669_2468_4f77_b3a4_0eea66f35352.slice/crio-21cb62ac8284af04cb9175256dd13fd95a9605e5ec29f53e1b3928af9ed5adf6 WatchSource:0}: Error finding container 21cb62ac8284af04cb9175256dd13fd95a9605e5ec29f53e1b3928af9ed5adf6: Status 404 returned error can't find the container with id 21cb62ac8284af04cb9175256dd13fd95a9605e5ec29f53e1b3928af9ed5adf6 Nov 03 20:03:46.427766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:46.427729 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:46.428179 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:46.428156 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:46.447465 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:46.447440 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:03:47.328611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:47.328563 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" event={"ID":"8f264669-2468-4f77-b3a4-0eea66f35352","Type":"ContainerStarted","Data":"9fb2b98610f7336e8bb0364be4765e00390cda566adecfa912dd55a987669bdf"} Nov 03 20:03:47.328611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:47.328614 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" event={"ID":"8f264669-2468-4f77-b3a4-0eea66f35352","Type":"ContainerStarted","Data":"21cb62ac8284af04cb9175256dd13fd95a9605e5ec29f53e1b3928af9ed5adf6"} Nov 03 20:03:47.346122 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:47.346073 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podStartSLOduration=2.346058713 podStartE2EDuration="2.346058713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:47.343938599 +0000 UTC m=+1457.306317038" watchObservedRunningTime="2026-11-03 20:03:47.346058713 +0000 UTC m=+1457.308437152" Nov 03 20:03:48.008400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:48.008365 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:51.883960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:51.883912 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:51.895590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:51.895555 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:51.934334 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:51.934272 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9"] Nov 03 20:03:51.939712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:51.939679 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:51.951860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:51.951825 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-11a5b0-openshift-default-dockercfg-jmpd9\"" Nov 03 20:03:51.960057 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:51.960004 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9"] Nov 03 20:03:52.009729 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.009689 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.009897 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.009820 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.010000 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.009984 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.010053 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.010031 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.010175 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.010104 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wktk\" (UniqueName: \"kubernetes.io/projected/8f26fe8b-ab5c-4978-ab59-d163324180ff-kube-api-access-2wktk\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.010233 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.010215 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.010301 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.010277 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f26fe8b-ab5c-4978-ab59-d163324180ff-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.010399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.010380 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.010494 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.010481 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.111478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.111434 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.111699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.111490 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.111699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.111577 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.111837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.111727 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.111837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.111752 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.111837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.111824 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wktk\" (UniqueName: \"kubernetes.io/projected/8f26fe8b-ab5c-4978-ab59-d163324180ff-kube-api-access-2wktk\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.111989 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.111878 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.111989 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.111932 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f26fe8b-ab5c-4978-ab59-d163324180ff-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.112095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.112035 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.112095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.112072 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.112234 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.112130 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.112305 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.112251 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.112367 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.112354 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.112685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.112667 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f26fe8b-ab5c-4978-ab59-d163324180ff-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.113940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.113913 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.114679 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.114661 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.119395 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.119365 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.119621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.119604 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wktk\" (UniqueName: \"kubernetes.io/projected/8f26fe8b-ab5c-4978-ab59-d163324180ff-kube-api-access-2wktk\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.277180 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.277085 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:52.459251 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.459217 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh"] Nov 03 20:03:52.464837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.464802 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.493769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.493727 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh"] Nov 03 20:03:52.570689 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.570661 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9"] Nov 03 20:03:52.572684 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:03:52.572657 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f26fe8b_ab5c_4978_ab59_d163324180ff.slice/crio-afd22a09ed7b78a1155395222cbc3f10e95714a7707ea1f9fcf0c64a43a004f3 WatchSource:0}: Error finding container afd22a09ed7b78a1155395222cbc3f10e95714a7707ea1f9fcf0c64a43a004f3: Status 404 returned error can't find the container with id afd22a09ed7b78a1155395222cbc3f10e95714a7707ea1f9fcf0c64a43a004f3 Nov 03 20:03:52.575280 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.575244 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:03:52.575400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.575324 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:03:52.575400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.575370 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:03:52.620624 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.620595 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.620792 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.620645 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.620792 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.620760 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.620885 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.620802 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.620885 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.620867 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.620950 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.620915 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1c43d3-728e-4c45-a379-d3ea36351718-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.621009 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.620993 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsvg2\" (UniqueName: \"kubernetes.io/projected/ff1c43d3-728e-4c45-a379-d3ea36351718-kube-api-access-rsvg2\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.621048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.621024 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.621086 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.621053 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.722455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.722416 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsvg2\" (UniqueName: \"kubernetes.io/projected/ff1c43d3-728e-4c45-a379-d3ea36351718-kube-api-access-rsvg2\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.722679 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.722473 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.722679 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.722514 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.722812 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.722678 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.722812 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.722740 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.723037 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.722998 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.723174 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.723057 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.723174 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.723059 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.723174 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.723115 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.723174 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.723004 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.723174 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.723144 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1c43d3-728e-4c45-a379-d3ea36351718-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.723437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.723250 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.723437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.723416 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.723936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.723907 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1c43d3-728e-4c45-a379-d3ea36351718-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.725590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.725566 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.725710 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.725695 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.730342 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.730317 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.730447 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.730400 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsvg2\" (UniqueName: \"kubernetes.io/projected/ff1c43d3-728e-4c45-a379-d3ea36351718-kube-api-access-rsvg2\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:52.792832 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:52.792791 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:53.083985 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:03:53.083954 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1c43d3_728e_4c45_a379_d3ea36351718.slice/crio-9d6cb6e94aae8d9666a48e2502e49df679becf32c92731feaabf21fd16696427 WatchSource:0}: Error finding container 9d6cb6e94aae8d9666a48e2502e49df679becf32c92731feaabf21fd16696427: Status 404 returned error can't find the container with id 9d6cb6e94aae8d9666a48e2502e49df679becf32c92731feaabf21fd16696427 Nov 03 20:03:53.085280 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:53.085252 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh"] Nov 03 20:03:53.086955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:53.086918 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:03:53.087042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:53.087003 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:03:53.087087 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:53.087052 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:03:53.404461 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:53.404368 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" event={"ID":"8f26fe8b-ab5c-4978-ab59-d163324180ff","Type":"ContainerStarted","Data":"8371bf3a77d805118c08fb37a0d865b00b7e22109a89f5b3d2821cb539b01fb4"} Nov 03 20:03:53.404461 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:53.404437 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" event={"ID":"8f26fe8b-ab5c-4978-ab59-d163324180ff","Type":"ContainerStarted","Data":"afd22a09ed7b78a1155395222cbc3f10e95714a7707ea1f9fcf0c64a43a004f3"} Nov 03 20:03:53.406677 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:53.406651 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" event={"ID":"ff1c43d3-728e-4c45-a379-d3ea36351718","Type":"ContainerStarted","Data":"3437d93556a9297a6648a0d5a825c01049bc8dc7be9eb2da1998aa0b83b83765"} Nov 03 20:03:53.406766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:53.406702 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" event={"ID":"ff1c43d3-728e-4c45-a379-d3ea36351718","Type":"ContainerStarted","Data":"9d6cb6e94aae8d9666a48e2502e49df679becf32c92731feaabf21fd16696427"} Nov 03 20:03:53.430561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:53.430467 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" podStartSLOduration=2.4304541029999998 podStartE2EDuration="2.430454103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:53.424436234 +0000 UTC m=+1463.386814673" watchObservedRunningTime="2026-11-03 20:03:53.430454103 +0000 UTC m=+1463.392832541" Nov 03 20:03:53.632579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:53.629334 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:53.671160 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:53.671067 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:53.692736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:53.692654 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" podStartSLOduration=1.692634059 podStartE2EDuration="1.692634059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:53.466911211 +0000 UTC m=+1463.429289651" watchObservedRunningTime="2026-11-03 20:03:53.692634059 +0000 UTC m=+1463.655012501" Nov 03 20:03:53.799572 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:53.798707 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:03:54.296557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:54.293461 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:03:54.296557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:54.295691 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:03:54.296557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:54.295756 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" podUID="8f26fe8b-ab5c-4978-ab59-d163324180ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 20:03:54.456588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:54.443453 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:54.801247 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:54.800784 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:03:54.801247 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:54.800866 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" podUID="ff1c43d3-728e-4c45-a379-d3ea36351718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 20:03:55.280992 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:55.280951 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:03:55.281295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:55.281264 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" podUID="8f26fe8b-ab5c-4978-ab59-d163324180ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 20:03:55.794005 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:55.793961 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:03:55.794665 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:55.794627 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" podUID="ff1c43d3-728e-4c45-a379-d3ea36351718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 20:03:56.013555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:56.011764 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:03:56.284563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:56.283978 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:03:56.284563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:56.284040 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" podUID="8f26fe8b-ab5c-4978-ab59-d163324180ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 20:03:56.438591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:56.435043 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:03:56.438591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:56.435815 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:56.467745 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:56.467700 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:56.498573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:56.490355 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:56.588617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:56.568919 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:56.669069 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:56.667172 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Nov 03 20:03:56.669774 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:56.669738 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:56.801582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:56.800998 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:03:56.801582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:56.801072 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" podUID="ff1c43d3-728e-4c45-a379-d3ea36351718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 20:03:56.856731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:56.856625 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:56.857320 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:56.857285 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:56.907513 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:56.907468 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:57.010750 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.010698 5111 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:57.283569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.281370 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:03:57.283569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.281442 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" podUID="8f26fe8b-ab5c-4978-ab59-d163324180ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 20:03:57.295937 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.294938 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:57.405557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.404984 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:57.439554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.439108 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:03:57.526593 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.470294 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:57.526593 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.487616 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 20:03:57.526593 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.517975 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:03:57.526593 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.518044 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:03:57.526593 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.518080 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:57.633409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.559179 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:57.647058 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.647028 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:03:57.692428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.692243 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 20:03:57.692428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.692371 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:57.725316 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.725280 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:03:57.729693 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.729663 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:03:57.852384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.852351 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:03:57.877658 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.853007 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" podUID="ff1c43d3-728e-4c45-a379-d3ea36351718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 20:03:57.881833 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.881793 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:03:57.921780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.921682 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:03:58.025759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.950872 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:58.025759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.951191 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:58.129376 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:57.881523 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:03:58.282620 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:58.272236 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:03:58.282926 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:58.282901 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:03:58.385034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:58.334615 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:03:58.385034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:58.334679 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:03:58.385034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:58.355874 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:03:58.385034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:58.359587 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:03:58.424555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:58.410421 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:03:58.428984 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:58.428945 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" podUID="8f26fe8b-ab5c-4978-ab59-d163324180ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 20:03:58.436563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:58.415452 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:03:58.471704 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:58.471671 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:03:58.471977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:58.471960 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:03:58.472112 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:58.472100 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:03:58.806628 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:58.806588 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:03:58.806939 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:58.806904 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" podUID="ff1c43d3-728e-4c45-a379-d3ea36351718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 20:03:59.283557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:59.283202 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:03:59.283557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:59.283266 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" podUID="8f26fe8b-ab5c-4978-ab59-d163324180ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 20:03:59.797556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:59.796919 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:03:59.797556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:03:59.796988 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" podUID="ff1c43d3-728e-4c45-a379-d3ea36351718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 20:04:00.281982 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:00.281940 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:04:00.282273 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:00.282237 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" podUID="8f26fe8b-ab5c-4978-ab59-d163324180ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 20:04:00.793626 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:00.793582 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:04:00.794160 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:00.793644 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" podUID="ff1c43d3-728e-4c45-a379-d3ea36351718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 20:04:02.279328 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:02.278558 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:04:02.279328 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:02.278640 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" podUID="8f26fe8b-ab5c-4978-ab59-d163324180ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:02.326733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:02.326697 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:04:02.327380 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:02.327354 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:04:02.328272 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:02.328251 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:04:02.794094 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:02.794055 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:04:02.794264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:02.794115 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" podUID="ff1c43d3-728e-4c45-a379-d3ea36351718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:02.838559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:02.838515 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:04:02.838941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:02.838908 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:04:02.842287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:02.842261 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:04:05.059574 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:04:05.059509 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:04:06.020821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:06.020791 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:04:06.032630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:06.032595 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:04:07.062986 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.062939 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw"] Nov 03 20:04:07.192277 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.192245 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw"] Nov 03 20:04:07.193276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.193260 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q"] Nov 03 20:04:07.194014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.193219 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.198455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.198437 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q"] Nov 03 20:04:07.198617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.198597 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.336895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.336798 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z"] Nov 03 20:04:07.342382 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.342355 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.354430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.354399 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z"] Nov 03 20:04:07.361171 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.361136 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653d951c-536d-405b-bc11-c9985924b511-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.361369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.361345 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653d951c-536d-405b-bc11-c9985924b511-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.361487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.361447 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.361602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.361577 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.361677 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.361659 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w7zp\" (UniqueName: \"kubernetes.io/projected/653d951c-536d-405b-bc11-c9985924b511-kube-api-access-9w7zp\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.361726 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.361702 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.361783 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.361742 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.361916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.361900 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.361993 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.361978 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.362048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.362006 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea4d795-91eb-4eed-9ce9-891d4d26337c-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.362145 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.362120 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.362200 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.362183 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653d951c-536d-405b-bc11-c9985924b511-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.362247 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.362226 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.362298 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.362274 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45jhd\" (UniqueName: \"kubernetes.io/projected/eea4d795-91eb-4eed-9ce9-891d4d26337c-kube-api-access-45jhd\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.362349 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.362311 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.362349 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.362336 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.362440 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.362364 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.362440 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.362394 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.463670 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.463628 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w7zp\" (UniqueName: \"kubernetes.io/projected/653d951c-536d-405b-bc11-c9985924b511-kube-api-access-9w7zp\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.463670 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.463671 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.463930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.463710 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.463930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.463744 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.463930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.463837 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.463930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.463902 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.464189 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464043 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.464189 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464077 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea4d795-91eb-4eed-9ce9-891d4d26337c-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.464189 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464139 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.464351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464192 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.464351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464213 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653d951c-536d-405b-bc11-c9985924b511-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.464351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464141 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.464351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464219 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.464351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464240 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.464351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464265 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.464351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464288 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.464742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464370 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.464742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464486 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45jhd\" (UniqueName: \"kubernetes.io/projected/eea4d795-91eb-4eed-9ce9-891d4d26337c-kube-api-access-45jhd\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.464742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464559 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.464742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464587 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.464742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464616 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.464742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464647 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.465023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464774 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5npf\" (UniqueName: \"kubernetes.io/projected/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-kube-api-access-b5npf\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.465023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464823 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653d951c-536d-405b-bc11-c9985924b511-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.465023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464825 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea4d795-91eb-4eed-9ce9-891d4d26337c-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.465023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464852 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653d951c-536d-405b-bc11-c9985924b511-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.465023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464879 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.465023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464912 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.465023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464964 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.465023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464974 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.465023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.464992 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.465023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.465021 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.465478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.465196 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.465478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.465269 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.465478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.465304 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.465736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.465712 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.465813 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.465716 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653d951c-536d-405b-bc11-c9985924b511-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.466901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.466874 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.467254 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.467238 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653d951c-536d-405b-bc11-c9985924b511-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.467635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.467612 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.467767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.467749 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.475877 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.475849 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w7zp\" (UniqueName: \"kubernetes.io/projected/653d951c-536d-405b-bc11-c9985924b511-kube-api-access-9w7zp\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.476835 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.476811 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45jhd\" (UniqueName: \"kubernetes.io/projected/eea4d795-91eb-4eed-9ce9-891d4d26337c-kube-api-access-45jhd\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.477066 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.477046 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.477131 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.477090 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653d951c-536d-405b-bc11-c9985924b511-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.515823 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.515789 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:07.526559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.526506 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:07.568016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.567352 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.568016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.567427 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.568016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.567459 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.568016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.567514 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.568016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.567644 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.568016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.567685 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.568016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.567840 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.568016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.567873 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.568522 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.568050 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5npf\" (UniqueName: \"kubernetes.io/projected/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-kube-api-access-b5npf\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.568606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.568582 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.569988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.568912 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.569988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.568922 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.569988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.568975 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.569988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.569657 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.571543 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.571393 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.572034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.572013 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.581414 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.581389 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.586165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.586139 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5npf\" (UniqueName: \"kubernetes.io/projected/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-kube-api-access-b5npf\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.664131 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.664092 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:07.820792 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.820754 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw"] Nov 03 20:04:07.822991 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:04:07.822187 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea4d795_91eb_4eed_9ce9_891d4d26337c.slice/crio-b7b23099cd2238bdbd7f09cfac84cce9b8896838d09484f7cc7c0c525d19f715 WatchSource:0}: Error finding container b7b23099cd2238bdbd7f09cfac84cce9b8896838d09484f7cc7c0c525d19f715: Status 404 returned error can't find the container with id b7b23099cd2238bdbd7f09cfac84cce9b8896838d09484f7cc7c0c525d19f715 Nov 03 20:04:07.825320 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.825286 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:07.825408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.825356 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:07.825408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.825390 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:07.827195 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:04:07.827157 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653d951c_536d_405b_bc11_c9985924b511.slice/crio-34de2bcea34147d5bff36f28aa9e9ef23e5132aa99e88fa95c1406d6ae76198a WatchSource:0}: Error finding container 34de2bcea34147d5bff36f28aa9e9ef23e5132aa99e88fa95c1406d6ae76198a: Status 404 returned error can't find the container with id 34de2bcea34147d5bff36f28aa9e9ef23e5132aa99e88fa95c1406d6ae76198a Nov 03 20:04:07.827966 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.827939 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q"] Nov 03 20:04:07.840918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.840849 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:07.841043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.840936 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:07.841043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.840978 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:07.905290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.905257 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" event={"ID":"653d951c-536d-405b-bc11-c9985924b511","Type":"ContainerStarted","Data":"34de2bcea34147d5bff36f28aa9e9ef23e5132aa99e88fa95c1406d6ae76198a"} Nov 03 20:04:07.907205 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.907147 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" event={"ID":"eea4d795-91eb-4eed-9ce9-891d4d26337c","Type":"ContainerStarted","Data":"b7b23099cd2238bdbd7f09cfac84cce9b8896838d09484f7cc7c0c525d19f715"} Nov 03 20:04:07.958869 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.958591 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z"] Nov 03 20:04:07.961935 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:04:07.961902 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cc3ca7_a54e_4172_92aa_3ffe98ee0b1f.slice/crio-3f08834b7ed8eff5f27b144e7ea59468d59666422c3a1a15ce499c7029e7a1c8 WatchSource:0}: Error finding container 3f08834b7ed8eff5f27b144e7ea59468d59666422c3a1a15ce499c7029e7a1c8: Status 404 returned error can't find the container with id 3f08834b7ed8eff5f27b144e7ea59468d59666422c3a1a15ce499c7029e7a1c8 Nov 03 20:04:07.971266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.966489 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:07.971266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.966601 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:07.971266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:07.966662 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:08.936042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:08.936007 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" event={"ID":"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f","Type":"ContainerStarted","Data":"5af8b136c308d412ed77327969f64975d358c108ce260bed9134c16312cb742f"} Nov 03 20:04:08.936559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:08.936061 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" event={"ID":"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f","Type":"ContainerStarted","Data":"3f08834b7ed8eff5f27b144e7ea59468d59666422c3a1a15ce499c7029e7a1c8"} Nov 03 20:04:08.938508 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:08.938483 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" event={"ID":"653d951c-536d-405b-bc11-c9985924b511","Type":"ContainerStarted","Data":"339047ed53a74a05fb461cae65cccaa9ad55246a4199addf4b697d1e56fda5c8"} Nov 03 20:04:08.940870 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:08.940851 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" event={"ID":"eea4d795-91eb-4eed-9ce9-891d4d26337c","Type":"ContainerStarted","Data":"c5234fa8702820db67bec68464d179e92f8b9eae792cb8795e5f71126c9ccfbf"} Nov 03 20:04:08.982837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:08.982770 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" podStartSLOduration=1.98274884 podStartE2EDuration="1.98274884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:08.963074263 +0000 UTC m=+1478.925452715" watchObservedRunningTime="2026-11-03 20:04:08.98274884 +0000 UTC m=+1478.945127279" Nov 03 20:04:09.037605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:09.030362 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" podStartSLOduration=2.030339092 podStartE2EDuration="2.030339092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:09.00480976 +0000 UTC m=+1478.967188198" watchObservedRunningTime="2026-11-03 20:04:09.030339092 +0000 UTC m=+1478.992717533" Nov 03 20:04:09.038880 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:09.038817 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" podStartSLOduration=2.038801563 podStartE2EDuration="2.038801563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:09.023805304 +0000 UTC m=+1478.986183746" watchObservedRunningTime="2026-11-03 20:04:09.038801563 +0000 UTC m=+1479.001180002" Nov 03 20:04:09.519640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:09.519604 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:09.520066 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:09.520040 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 20:04:09.520221 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:09.520197 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" podUID="eea4d795-91eb-4eed-9ce9-891d4d26337c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 20:04:09.530564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:09.529638 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:09.530564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:09.529980 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 20:04:09.530564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:09.530025 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" podUID="653d951c-536d-405b-bc11-c9985924b511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 20:04:09.670595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:09.666641 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:09.670595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:09.667623 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 20:04:09.670595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:09.667689 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" podUID="96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 20:04:10.088485 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.088445 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm"] Nov 03 20:04:10.134876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.134845 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.136939 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.136905 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm"] Nov 03 20:04:10.139893 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.139874 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 20:04:10.140014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.139920 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 20:04:10.140014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.139985 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gjlpz\"" Nov 03 20:04:10.140014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.139871 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 20:04:10.321386 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.321324 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d0a532e-8921-486e-ad83-2cb3c1e72292-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.321739 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.321710 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.321949 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.321928 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.322107 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.322090 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.322311 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.322294 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.322447 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.322431 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.322643 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.322625 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jfkp\" (UniqueName: \"kubernetes.io/projected/7d0a532e-8921-486e-ad83-2cb3c1e72292-kube-api-access-5jfkp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.427560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.427501 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d0a532e-8921-486e-ad83-2cb3c1e72292-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.429137 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.429109 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.434489 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.434464 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.434730 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.434713 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.434929 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.434916 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.435033 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.435019 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.435175 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.435160 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jfkp\" (UniqueName: \"kubernetes.io/projected/7d0a532e-8921-486e-ad83-2cb3c1e72292-kube-api-access-5jfkp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.435789 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.430027 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.439024 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.438988 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.439238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.439218 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d0a532e-8921-486e-ad83-2cb3c1e72292-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.442868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.440060 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.442868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.441817 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.479309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.479269 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.503579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.496365 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jfkp\" (UniqueName: \"kubernetes.io/projected/7d0a532e-8921-486e-ad83-2cb3c1e72292-kube-api-access-5jfkp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vk8mm\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.518242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.518205 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 20:04:10.518555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.518507 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" podUID="eea4d795-91eb-4eed-9ce9-891d4d26337c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 20:04:10.534596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.534544 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 20:04:10.534964 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.534933 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" podUID="653d951c-536d-405b-bc11-c9985924b511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 20:04:10.535376 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.535303 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:10.682911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.679209 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 20:04:10.682911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:10.679272 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" podUID="96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 20:04:11.343095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:11.343062 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm"] Nov 03 20:04:11.375119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:11.375086 5111 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:04:11.537298 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:11.537263 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 20:04:11.562557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:11.537561 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 20:04:11.562557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:11.539058 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" podUID="653d951c-536d-405b-bc11-c9985924b511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 20:04:11.562557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:11.537636 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" podUID="eea4d795-91eb-4eed-9ce9-891d4d26337c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 20:04:11.737560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:11.734840 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 20:04:11.737560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:11.734909 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" podUID="96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 20:04:12.326602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:12.326559 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" event={"ID":"7d0a532e-8921-486e-ad83-2cb3c1e72292","Type":"ContainerStarted","Data":"444f097ce7af8bc8bc3333089ecc424ebe4745134bc0bce4e800d147a50f75f7"} Nov 03 20:04:12.543692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:12.543597 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 20:04:12.543692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:12.543659 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" podUID="653d951c-536d-405b-bc11-c9985924b511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 20:04:12.578623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:12.558346 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 20:04:12.578623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:12.558416 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" podUID="eea4d795-91eb-4eed-9ce9-891d4d26337c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 20:04:12.674736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:12.674694 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 20:04:12.678555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:12.675752 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" podUID="96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 20:04:13.521866 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:13.521826 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 20:04:13.522200 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:13.522168 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" podUID="eea4d795-91eb-4eed-9ce9-891d4d26337c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 20:04:13.566958 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:13.566918 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 20:04:13.583544 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:13.583491 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" podUID="653d951c-536d-405b-bc11-c9985924b511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 20:04:13.742757 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:13.742701 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 20:04:13.743027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:13.742997 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" podUID="96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 20:04:14.516979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:14.516596 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 20:04:14.516979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:14.516655 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" podUID="eea4d795-91eb-4eed-9ce9-891d4d26337c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 20:04:14.527108 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:14.526839 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 20:04:14.527108 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:14.526888 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" podUID="653d951c-536d-405b-bc11-c9985924b511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 20:04:14.666333 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:14.666293 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 20:04:14.666873 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:14.666369 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" podUID="96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 20:04:15.571221 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:15.569231 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:15.571221 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:15.570900 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:15.579115 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:15.576101 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:04:15.842119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:15.840325 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:15.842119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:15.841854 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:15.843121 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:15.843102 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:04:16.074102 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:16.074066 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:16.075046 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:16.075024 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:16.076055 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:16.076034 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:04:16.958110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:16.958072 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:16.958390 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:16.958161 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:17.438233 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:17.438191 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" event={"ID":"7d0a532e-8921-486e-ad83-2cb3c1e72292","Type":"ContainerStarted","Data":"b0c371317c1fea9528ae424c265322c2c66e962a6528c1a93b03014210a9da0f"} Nov 03 20:04:18.447504 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:18.447472 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:18.449124 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:18.449100 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:04:18.481210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:18.481100 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" podStartSLOduration=2.969989091 podStartE2EDuration="8.48107766s" totalImagesPullingTime="5.511088569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:17.461734025 +0000 UTC m=+1487.424112464" watchObservedRunningTime="2026-11-03 20:04:18.48107766 +0000 UTC m=+1488.443456099" Nov 03 20:04:21.408855 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:21.407976 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:21.422707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:21.421626 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:21.422707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:21.421987 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:04:22.227699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:22.227650 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:30.429105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:30.429072 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:04:30.434300 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:30.434270 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:04:30.444353 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:30.444285 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:04:30.446290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:30.446264 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:04:33.671345 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.671301 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz"] Nov 03 20:04:33.677398 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.677373 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz" Nov 03 20:04:33.680928 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.680899 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:04:33.682080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.682053 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:04:33.682080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.682074 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:04:33.682258 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.682092 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:04:33.684163 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.684137 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz"] Nov 03 20:04:33.755466 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.755421 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqc8m\" (UniqueName: \"kubernetes.io/projected/cb95e881-d3b3-4b48-9242-c05140b94826-kube-api-access-dqc8m\") pod \"maas-api-revoke-sub-2514ae91-28pwz\" (UID: \"cb95e881-d3b3-4b48-9242-c05140b94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz" Nov 03 20:04:33.755697 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.755619 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb95e881-d3b3-4b48-9242-c05140b94826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2514ae91-28pwz\" (UID: \"cb95e881-d3b3-4b48-9242-c05140b94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz" Nov 03 20:04:33.755980 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.755947 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb95e881-d3b3-4b48-9242-c05140b94826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2514ae91-28pwz\" (UID: \"cb95e881-d3b3-4b48-9242-c05140b94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz" Nov 03 20:04:33.857205 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.857167 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb95e881-d3b3-4b48-9242-c05140b94826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2514ae91-28pwz\" (UID: \"cb95e881-d3b3-4b48-9242-c05140b94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz" Nov 03 20:04:33.857399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.857372 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb95e881-d3b3-4b48-9242-c05140b94826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2514ae91-28pwz\" (UID: \"cb95e881-d3b3-4b48-9242-c05140b94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz" Nov 03 20:04:33.857450 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.857436 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqc8m\" (UniqueName: \"kubernetes.io/projected/cb95e881-d3b3-4b48-9242-c05140b94826-kube-api-access-dqc8m\") pod \"maas-api-revoke-sub-2514ae91-28pwz\" (UID: \"cb95e881-d3b3-4b48-9242-c05140b94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz" Nov 03 20:04:33.858095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.858066 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb95e881-d3b3-4b48-9242-c05140b94826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2514ae91-28pwz\" (UID: \"cb95e881-d3b3-4b48-9242-c05140b94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz" Nov 03 20:04:33.859758 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.859733 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb95e881-d3b3-4b48-9242-c05140b94826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2514ae91-28pwz\" (UID: \"cb95e881-d3b3-4b48-9242-c05140b94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz" Nov 03 20:04:33.865633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.865613 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqc8m\" (UniqueName: \"kubernetes.io/projected/cb95e881-d3b3-4b48-9242-c05140b94826-kube-api-access-dqc8m\") pod \"maas-api-revoke-sub-2514ae91-28pwz\" (UID: \"cb95e881-d3b3-4b48-9242-c05140b94826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz" Nov 03 20:04:33.996085 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:33.995992 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz" Nov 03 20:04:34.287550 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:34.287496 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz"] Nov 03 20:04:34.288574 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:04:34.288543 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb95e881_d3b3_4b48_9242_c05140b94826.slice/crio-b103cb7f91608d08cd2896ac67c37d3d3a6a10710be7eedade83ce33512d814b WatchSource:0}: Error finding container b103cb7f91608d08cd2896ac67c37d3d3a6a10710be7eedade83ce33512d814b: Status 404 returned error can't find the container with id b103cb7f91608d08cd2896ac67c37d3d3a6a10710be7eedade83ce33512d814b Nov 03 20:04:34.663198 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:34.663161 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz" event={"ID":"cb95e881-d3b3-4b48-9242-c05140b94826","Type":"ContainerStarted","Data":"b103cb7f91608d08cd2896ac67c37d3d3a6a10710be7eedade83ce33512d814b"} Nov 03 20:04:35.058306 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:04:35.058210 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:04:35.674199 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:35.674163 5111 generic.go:356] "Generic (PLEG): container finished" podID="cb95e881-d3b3-4b48-9242-c05140b94826" containerID="28cfe07dd097152ebfa0e067e51b02b5c0332d109f463c9f5e47e0687f224994" exitCode=0 Nov 03 20:04:35.674420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:35.674251 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz" event={"ID":"cb95e881-d3b3-4b48-9242-c05140b94826","Type":"ContainerDied","Data":"28cfe07dd097152ebfa0e067e51b02b5c0332d109f463c9f5e47e0687f224994"} Nov 03 20:04:36.888022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:36.887999 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz" Nov 03 20:04:37.019777 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.019684 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb95e881-d3b3-4b48-9242-c05140b94826-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb95e881-d3b3-4b48-9242-c05140b94826-maas-api-openshift-ca\") pod \"cb95e881-d3b3-4b48-9242-c05140b94826\" (UID: \"cb95e881-d3b3-4b48-9242-c05140b94826\") " Nov 03 20:04:37.020073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.020048 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb95e881-d3b3-4b48-9242-c05140b94826-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb95e881-d3b3-4b48-9242-c05140b94826-maas-api-xks-ca\") pod \"cb95e881-d3b3-4b48-9242-c05140b94826\" (UID: \"cb95e881-d3b3-4b48-9242-c05140b94826\") " Nov 03 20:04:37.020797 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.020770 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb95e881-d3b3-4b48-9242-c05140b94826-kube-api-access-dqc8m\" (UniqueName: \"kubernetes.io/projected/cb95e881-d3b3-4b48-9242-c05140b94826-kube-api-access-dqc8m\") pod \"cb95e881-d3b3-4b48-9242-c05140b94826\" (UID: \"cb95e881-d3b3-4b48-9242-c05140b94826\") " Nov 03 20:04:37.020907 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.020788 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb95e881-d3b3-4b48-9242-c05140b94826-maas-api-openshift-ca" pod "cb95e881-d3b3-4b48-9242-c05140b94826" (UID: "cb95e881-d3b3-4b48-9242-c05140b94826"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:37.021338 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.021317 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb95e881-d3b3-4b48-9242-c05140b94826-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:37.023858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.023821 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb95e881-d3b3-4b48-9242-c05140b94826-maas-api-xks-ca" pod "cb95e881-d3b3-4b48-9242-c05140b94826" (UID: "cb95e881-d3b3-4b48-9242-c05140b94826"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:04:37.024104 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.023835 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb95e881-d3b3-4b48-9242-c05140b94826-kube-api-access-dqc8m" pod "cb95e881-d3b3-4b48-9242-c05140b94826" (UID: "cb95e881-d3b3-4b48-9242-c05140b94826"). InnerVolumeSpecName "kube-api-access-dqc8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:37.122285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.122247 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb95e881-d3b3-4b48-9242-c05140b94826-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:37.122285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.122282 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqc8m\" (UniqueName: \"kubernetes.io/projected/cb95e881-d3b3-4b48-9242-c05140b94826-kube-api-access-dqc8m\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:37.306987 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.306902 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2"] Nov 03 20:04:37.307577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.307558 5111 cpu_manager.go:414] "containerMap: removing container" podUID="cb95e881-d3b3-4b48-9242-c05140b94826" containerName="revoke-keys" Nov 03 20:04:37.307577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.307575 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb95e881-d3b3-4b48-9242-c05140b94826" containerName="revoke-keys" Nov 03 20:04:37.307740 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.307702 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb95e881-d3b3-4b48-9242-c05140b94826" containerName="revoke-keys" Nov 03 20:04:37.311409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.311384 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.334056 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.334019 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2"] Nov 03 20:04:37.427582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.427550 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.427793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.427600 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.427793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.427624 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.427923 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.427840 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f87478-ca01-41de-b602-f5a831e07913-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.427995 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.427961 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9lpx\" (UniqueName: \"kubernetes.io/projected/c1f87478-ca01-41de-b602-f5a831e07913-kube-api-access-g9lpx\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.428236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.428041 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.428236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.428104 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f87478-ca01-41de-b602-f5a831e07913-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.428236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.428221 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.428430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.428313 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f87478-ca01-41de-b602-f5a831e07913-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.495317 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.495284 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj"] Nov 03 20:04:37.500269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.500238 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.523098 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.523060 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj"] Nov 03 20:04:37.535123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.532751 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.535123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.532825 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.535123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.532945 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.535123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.533063 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f87478-ca01-41de-b602-f5a831e07913-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.535123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.533148 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9lpx\" (UniqueName: \"kubernetes.io/projected/c1f87478-ca01-41de-b602-f5a831e07913-kube-api-access-g9lpx\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.535123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.533199 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.535123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.533263 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f87478-ca01-41de-b602-f5a831e07913-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.535123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.533307 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.535123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.533350 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f87478-ca01-41de-b602-f5a831e07913-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.536211 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.536180 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.536455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.536435 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.537105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.537075 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f87478-ca01-41de-b602-f5a831e07913-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.537206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.537102 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.537408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.537383 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.551222 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.540260 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f87478-ca01-41de-b602-f5a831e07913-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.556131 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.555913 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.556264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.556240 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f87478-ca01-41de-b602-f5a831e07913-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.556347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.556331 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9lpx\" (UniqueName: \"kubernetes.io/projected/c1f87478-ca01-41de-b602-f5a831e07913-kube-api-access-g9lpx\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.635267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.635227 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dntc\" (UniqueName: \"kubernetes.io/projected/8edc2022-355d-4c8e-9564-f2c0a2c6b397-kube-api-access-2dntc\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.635437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.635352 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.635437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.635389 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.635437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.635413 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.635592 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.635437 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.635592 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.635557 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.635759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.635737 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.635795 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.635775 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.635855 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.635843 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.662580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.662547 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:37.690835 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.690808 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz" Nov 03 20:04:37.690979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.690843 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz" event={"ID":"cb95e881-d3b3-4b48-9242-c05140b94826","Type":"ContainerDied","Data":"b103cb7f91608d08cd2896ac67c37d3d3a6a10710be7eedade83ce33512d814b"} Nov 03 20:04:37.690979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.690914 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b103cb7f91608d08cd2896ac67c37d3d3a6a10710be7eedade83ce33512d814b" Nov 03 20:04:37.738346 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.738308 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.738487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.738368 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.738487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.738427 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.738805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.738524 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dntc\" (UniqueName: \"kubernetes.io/projected/8edc2022-355d-4c8e-9564-f2c0a2c6b397-kube-api-access-2dntc\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.738805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.738616 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.738805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.738654 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.738805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.738687 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.738805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.738726 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.738805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.738791 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.739365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.739339 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.741620 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.740913 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.741620 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.740973 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.741620 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.741489 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.741620 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.741520 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.746524 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.746491 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.748050 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.748023 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.748330 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.748279 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.748680 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.748657 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dntc\" (UniqueName: \"kubernetes.io/projected/8edc2022-355d-4c8e-9564-f2c0a2c6b397-kube-api-access-2dntc\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.837200 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.836899 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:37.966217 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.966182 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2"] Nov 03 20:04:37.970279 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:04:37.970251 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f87478_ca01_41de_b602_f5a831e07913.slice/crio-51933282ba8d81b2eb68a274cfde3d157721a637069736024df1e28a7bce9392 WatchSource:0}: Error finding container 51933282ba8d81b2eb68a274cfde3d157721a637069736024df1e28a7bce9392: Status 404 returned error can't find the container with id 51933282ba8d81b2eb68a274cfde3d157721a637069736024df1e28a7bce9392 Nov 03 20:04:37.973027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.972988 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:37.973244 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.973220 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:37.973396 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:37.973374 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:38.159267 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:04:38.158581 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edc2022_355d_4c8e_9564_f2c0a2c6b397.slice/crio-f3756c1cd5e3971427c1c8937f9a9cf1dc0a8d6594721a2869959919612eda0a WatchSource:0}: Error finding container f3756c1cd5e3971427c1c8937f9a9cf1dc0a8d6594721a2869959919612eda0a: Status 404 returned error can't find the container with id f3756c1cd5e3971427c1c8937f9a9cf1dc0a8d6594721a2869959919612eda0a Nov 03 20:04:38.159267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:38.158946 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj"] Nov 03 20:04:38.161878 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:38.161839 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:38.161984 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:38.161925 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:38.161984 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:38.161965 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:04:38.701892 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:38.701845 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" event={"ID":"8edc2022-355d-4c8e-9564-f2c0a2c6b397","Type":"ContainerStarted","Data":"6bca2a51b6953dbbd9c11fd32777b34f6b955e3ff0dd1b6761d2dd5a21d4b555"} Nov 03 20:04:38.702084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:38.701911 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" event={"ID":"8edc2022-355d-4c8e-9564-f2c0a2c6b397","Type":"ContainerStarted","Data":"f3756c1cd5e3971427c1c8937f9a9cf1dc0a8d6594721a2869959919612eda0a"} Nov 03 20:04:38.703675 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:38.703646 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" event={"ID":"c1f87478-ca01-41de-b602-f5a831e07913","Type":"ContainerStarted","Data":"6deebfa3bdcf1565192a7b61b9474c838a90a90004bb0239c66e59bebb122765"} Nov 03 20:04:38.703805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:38.703695 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" event={"ID":"c1f87478-ca01-41de-b602-f5a831e07913","Type":"ContainerStarted","Data":"51933282ba8d81b2eb68a274cfde3d157721a637069736024df1e28a7bce9392"} Nov 03 20:04:38.735324 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:38.732090 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" podStartSLOduration=1.732075292 podStartE2EDuration="1.732075292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:38.727895359 +0000 UTC m=+1508.690273798" watchObservedRunningTime="2026-11-03 20:04:38.732075292 +0000 UTC m=+1508.694453730" Nov 03 20:04:38.762470 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:38.762390 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" podStartSLOduration=1.762365216 podStartE2EDuration="1.762365216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:38.749060549 +0000 UTC m=+1508.711439001" watchObservedRunningTime="2026-11-03 20:04:38.762365216 +0000 UTC m=+1508.724743659" Nov 03 20:04:38.837828 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:38.837794 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:39.663945 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:39.663907 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:39.668699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:39.668663 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:04:39.668981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:39.668953 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" podUID="c1f87478-ca01-41de-b602-f5a831e07913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 20:04:39.839952 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:39.839896 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:04:39.842703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:39.841244 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" podUID="8edc2022-355d-4c8e-9564-f2c0a2c6b397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 20:04:40.663060 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:40.663020 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:04:40.663493 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:40.663451 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" podUID="c1f87478-ca01-41de-b602-f5a831e07913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 20:04:40.837754 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:40.837725 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:04:40.838221 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:40.838196 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" podUID="8edc2022-355d-4c8e-9564-f2c0a2c6b397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 20:04:41.667556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:41.666984 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:04:41.667556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:41.667050 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" podUID="c1f87478-ca01-41de-b602-f5a831e07913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 20:04:41.839361 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:41.838835 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:04:41.839361 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:41.838900 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" podUID="8edc2022-355d-4c8e-9564-f2c0a2c6b397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 20:04:42.668561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:42.667704 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:04:42.668561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:42.667770 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" podUID="c1f87478-ca01-41de-b602-f5a831e07913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 20:04:42.838575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:42.838356 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:04:42.838575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:42.838411 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" podUID="8edc2022-355d-4c8e-9564-f2c0a2c6b397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 20:04:43.664557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:43.663955 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:04:43.664557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:43.664021 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" podUID="c1f87478-ca01-41de-b602-f5a831e07913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 20:04:43.837994 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:43.837948 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:04:43.838212 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:43.838025 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" podUID="8edc2022-355d-4c8e-9564-f2c0a2c6b397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 20:04:44.666557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:44.663861 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:04:44.666557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:44.663930 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" podUID="c1f87478-ca01-41de-b602-f5a831e07913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 20:04:44.838046 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:44.838006 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:04:44.838240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:44.838076 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" podUID="8edc2022-355d-4c8e-9564-f2c0a2c6b397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 20:04:45.745289 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:45.745251 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:45.745964 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:45.745938 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:45.747104 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:45.747074 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:04:46.092149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:46.092067 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:46.093154 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:46.093131 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:46.093902 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:46.093882 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:04:50.336555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:50.335631 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:50.361664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:50.361615 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:50.425581 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:50.416624 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:50.425931 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:50.425892 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:50.426630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:50.426604 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:50.427027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:50.427003 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:50.427545 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:50.427498 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:50.459704 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:50.459653 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:51.026054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.026008 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:51.275679 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.275628 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:51.337742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.337638 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:51.338429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.338299 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 20:04:51.338958 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.338934 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:04:51.371660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.371603 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:51.373372 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.373346 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:04:51.468557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.458562 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:04:51.468557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.459187 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:04:51.468557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.460906 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:51.468988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.468958 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:04:51.469416 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.469385 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:51.469833 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.469808 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:04:51.470225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.470199 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:04:51.491556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.484241 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:04:51.491556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.484307 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:04:51.491556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.484345 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:04:51.491556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.484382 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:04:51.491556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:51.484416 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:04:52.161554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:52.044438 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 20:04:52.161554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:52.045003 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:04:52.231398 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:52.231355 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:52.284288 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:52.283631 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 20:04:52.356560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:52.353486 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:52.392049 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:52.383339 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:04:52.392049 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:52.384442 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:04:52.466744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:52.466638 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:52.494593 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:52.480888 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:52.553292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:52.494892 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:52.554723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:52.494933 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:52.555159 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:52.494969 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:52.556323 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:52.495008 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:52.923691 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:52.923639 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:52.959698 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:52.959648 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:53.094560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:53.069936 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:53.309618 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:53.284359 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 20:04:53.354617 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:04:53.322715 5111 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.078s" Nov 03 20:04:53.356185 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:53.356154 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:04:53.356467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:53.356451 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:04:53.412587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:53.400685 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:04:53.412587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:53.401283 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:04:53.445588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:53.423480 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 20:04:53.534799 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:53.482848 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:04:53.572039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:53.571921 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:04:53.623906 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:53.623844 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 20:04:53.656251 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:53.624323 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:04:53.682906 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:53.624551 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:04:53.683898 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:53.625365 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:54.142558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.129809 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 20:04:54.323945 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.323732 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:54.410622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.405074 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:54.411021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.405647 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:54.434918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.434697 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:54.483512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.483382 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:04:54.548230 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.548184 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:04:54.574085 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.574053 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:04:54.575207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.575186 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:04:54.625621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.577515 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:04:54.644630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.628013 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:04:54.645059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.645036 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:04:54.647302 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.647282 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:04:54.654391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.654370 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:04:54.655681 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.655662 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm"] Nov 03 20:04:54.664684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.663462 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:04:54.692408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.692381 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm"] Nov 03 20:04:54.692695 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.692681 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:04:54.693735 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.693714 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" Nov 03 20:04:54.700766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.700723 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:04:54.701170 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.701139 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:04:54.701408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.701379 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:04:54.701583 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.701560 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:04:54.701749 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.701733 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:04:54.740112 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.740077 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:04:54.740291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.740151 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:04:54.857553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.856955 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af6db2e-d6c9-47fd-a41e-c412988701d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5c86072-j86mm\" (UID: \"1af6db2e-d6c9-47fd-a41e-c412988701d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" Nov 03 20:04:54.857553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.857076 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdv7j\" (UniqueName: \"kubernetes.io/projected/1af6db2e-d6c9-47fd-a41e-c412988701d7-kube-api-access-pdv7j\") pod \"maas-api-revoke-sub-c5c86072-j86mm\" (UID: \"1af6db2e-d6c9-47fd-a41e-c412988701d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" Nov 03 20:04:54.857553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.857515 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af6db2e-d6c9-47fd-a41e-c412988701d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5c86072-j86mm\" (UID: \"1af6db2e-d6c9-47fd-a41e-c412988701d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" Nov 03 20:04:54.959090 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.958997 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdv7j\" (UniqueName: \"kubernetes.io/projected/1af6db2e-d6c9-47fd-a41e-c412988701d7-kube-api-access-pdv7j\") pod \"maas-api-revoke-sub-c5c86072-j86mm\" (UID: \"1af6db2e-d6c9-47fd-a41e-c412988701d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" Nov 03 20:04:54.959361 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.959334 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af6db2e-d6c9-47fd-a41e-c412988701d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5c86072-j86mm\" (UID: \"1af6db2e-d6c9-47fd-a41e-c412988701d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" Nov 03 20:04:54.959465 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.959423 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af6db2e-d6c9-47fd-a41e-c412988701d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5c86072-j86mm\" (UID: \"1af6db2e-d6c9-47fd-a41e-c412988701d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" Nov 03 20:04:54.960033 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.960009 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af6db2e-d6c9-47fd-a41e-c412988701d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5c86072-j86mm\" (UID: \"1af6db2e-d6c9-47fd-a41e-c412988701d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" Nov 03 20:04:54.962563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.962522 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af6db2e-d6c9-47fd-a41e-c412988701d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5c86072-j86mm\" (UID: \"1af6db2e-d6c9-47fd-a41e-c412988701d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" Nov 03 20:04:54.973438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:54.973410 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdv7j\" (UniqueName: \"kubernetes.io/projected/1af6db2e-d6c9-47fd-a41e-c412988701d7-kube-api-access-pdv7j\") pod \"maas-api-revoke-sub-c5c86072-j86mm\" (UID: \"1af6db2e-d6c9-47fd-a41e-c412988701d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" Nov 03 20:04:55.062105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:55.062061 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" Nov 03 20:04:55.143102 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:55.143066 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9"] Nov 03 20:04:55.163370 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:55.157043 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9"] Nov 03 20:04:55.163370 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:55.157065 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" Nov 03 20:04:55.270915 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:55.270015 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5vbk\" (UniqueName: \"kubernetes.io/projected/e478b9aa-0913-49f0-82de-15d52734181f-kube-api-access-n5vbk\") pod \"maas-api-revoke-sub-ca5d1e66-xvvs9\" (UID: \"e478b9aa-0913-49f0-82de-15d52734181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" Nov 03 20:04:55.270915 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:55.270146 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e478b9aa-0913-49f0-82de-15d52734181f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca5d1e66-xvvs9\" (UID: \"e478b9aa-0913-49f0-82de-15d52734181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" Nov 03 20:04:55.270915 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:55.270201 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e478b9aa-0913-49f0-82de-15d52734181f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca5d1e66-xvvs9\" (UID: \"e478b9aa-0913-49f0-82de-15d52734181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" Nov 03 20:04:55.395558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:55.395319 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5vbk\" (UniqueName: \"kubernetes.io/projected/e478b9aa-0913-49f0-82de-15d52734181f-kube-api-access-n5vbk\") pod \"maas-api-revoke-sub-ca5d1e66-xvvs9\" (UID: \"e478b9aa-0913-49f0-82de-15d52734181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" Nov 03 20:04:55.395558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:55.395460 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e478b9aa-0913-49f0-82de-15d52734181f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca5d1e66-xvvs9\" (UID: \"e478b9aa-0913-49f0-82de-15d52734181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" Nov 03 20:04:55.395558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:55.395522 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e478b9aa-0913-49f0-82de-15d52734181f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca5d1e66-xvvs9\" (UID: \"e478b9aa-0913-49f0-82de-15d52734181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" Nov 03 20:04:55.415244 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:55.396481 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e478b9aa-0913-49f0-82de-15d52734181f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca5d1e66-xvvs9\" (UID: \"e478b9aa-0913-49f0-82de-15d52734181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" Nov 03 20:04:55.426561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:55.422302 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5vbk\" (UniqueName: \"kubernetes.io/projected/e478b9aa-0913-49f0-82de-15d52734181f-kube-api-access-n5vbk\") pod \"maas-api-revoke-sub-ca5d1e66-xvvs9\" (UID: \"e478b9aa-0913-49f0-82de-15d52734181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" Nov 03 20:04:55.426561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:55.425945 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e478b9aa-0913-49f0-82de-15d52734181f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca5d1e66-xvvs9\" (UID: \"e478b9aa-0913-49f0-82de-15d52734181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" Nov 03 20:04:55.537488 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:55.537418 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm"] Nov 03 20:04:55.593900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:55.580970 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" Nov 03 20:04:56.091959 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:04:56.091911 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode478b9aa_0913_49f0_82de_15d52734181f.slice/crio-c0f325ba8fbaee03b5e0c4cfd5bb06ae0e486083c7560c5f2e89614d2ea2c4ae WatchSource:0}: Error finding container c0f325ba8fbaee03b5e0c4cfd5bb06ae0e486083c7560c5f2e89614d2ea2c4ae: Status 404 returned error can't find the container with id c0f325ba8fbaee03b5e0c4cfd5bb06ae0e486083c7560c5f2e89614d2ea2c4ae Nov 03 20:04:56.094104 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:56.094070 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9"] Nov 03 20:04:56.651155 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:56.651112 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" event={"ID":"1af6db2e-d6c9-47fd-a41e-c412988701d7","Type":"ContainerStarted","Data":"4160924f0932ab43a0cb0d1eacd5528bf0aa63d5d80673ca08ee9ac14c0956fd"} Nov 03 20:04:56.651670 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:56.651173 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" event={"ID":"1af6db2e-d6c9-47fd-a41e-c412988701d7","Type":"ContainerStarted","Data":"3e7081cc525eb4ecd106d429ffa4d987e3668acb8d0533c5dea1649f70927ab3"} Nov 03 20:04:56.653197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:56.653167 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" event={"ID":"e478b9aa-0913-49f0-82de-15d52734181f","Type":"ContainerStarted","Data":"6198d922a2e60c32d3f21bf4530b4fc983738185fa592ee45c222c6b8dc94724"} Nov 03 20:04:56.653325 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:56.653221 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" event={"ID":"e478b9aa-0913-49f0-82de-15d52734181f","Type":"ContainerStarted","Data":"c0f325ba8fbaee03b5e0c4cfd5bb06ae0e486083c7560c5f2e89614d2ea2c4ae"} Nov 03 20:04:56.670456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:56.670405 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" podStartSLOduration=4.670391641 podStartE2EDuration="4.670391641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:56.667728824 +0000 UTC m=+1526.630107265" watchObservedRunningTime="2026-11-03 20:04:56.670391641 +0000 UTC m=+1526.632770110" Nov 03 20:04:56.686078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:56.686024 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" podStartSLOduration=1.68601006 podStartE2EDuration="1.68601006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:56.68494214 +0000 UTC m=+1526.647320610" watchObservedRunningTime="2026-11-03 20:04:56.68601006 +0000 UTC m=+1526.648388499" Nov 03 20:04:57.664154 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:57.664120 5111 generic.go:356] "Generic (PLEG): container finished" podID="e478b9aa-0913-49f0-82de-15d52734181f" containerID="6198d922a2e60c32d3f21bf4530b4fc983738185fa592ee45c222c6b8dc94724" exitCode=0 Nov 03 20:04:57.664672 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:57.664210 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" event={"ID":"e478b9aa-0913-49f0-82de-15d52734181f","Type":"ContainerDied","Data":"6198d922a2e60c32d3f21bf4530b4fc983738185fa592ee45c222c6b8dc94724"} Nov 03 20:04:57.666080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:57.666042 5111 generic.go:356] "Generic (PLEG): container finished" podID="1af6db2e-d6c9-47fd-a41e-c412988701d7" containerID="4160924f0932ab43a0cb0d1eacd5528bf0aa63d5d80673ca08ee9ac14c0956fd" exitCode=0 Nov 03 20:04:57.666202 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:57.666101 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" event={"ID":"1af6db2e-d6c9-47fd-a41e-c412988701d7","Type":"ContainerDied","Data":"4160924f0932ab43a0cb0d1eacd5528bf0aa63d5d80673ca08ee9ac14c0956fd"} Nov 03 20:04:58.892440 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:58.892414 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" Nov 03 20:04:58.896123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:58.896098 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" Nov 03 20:04:58.964366 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:58.964266 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1af6db2e-d6c9-47fd-a41e-c412988701d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af6db2e-d6c9-47fd-a41e-c412988701d7-maas-api-xks-ca\") pod \"1af6db2e-d6c9-47fd-a41e-c412988701d7\" (UID: \"1af6db2e-d6c9-47fd-a41e-c412988701d7\") " Nov 03 20:04:58.964576 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:58.964371 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af6db2e-d6c9-47fd-a41e-c412988701d7-kube-api-access-pdv7j\" (UniqueName: \"kubernetes.io/projected/1af6db2e-d6c9-47fd-a41e-c412988701d7-kube-api-access-pdv7j\") pod \"1af6db2e-d6c9-47fd-a41e-c412988701d7\" (UID: \"1af6db2e-d6c9-47fd-a41e-c412988701d7\") " Nov 03 20:04:58.964576 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:58.964416 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1af6db2e-d6c9-47fd-a41e-c412988701d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af6db2e-d6c9-47fd-a41e-c412988701d7-maas-api-openshift-ca\") pod \"1af6db2e-d6c9-47fd-a41e-c412988701d7\" (UID: \"1af6db2e-d6c9-47fd-a41e-c412988701d7\") " Nov 03 20:04:58.964731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:58.964579 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e478b9aa-0913-49f0-82de-15d52734181f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e478b9aa-0913-49f0-82de-15d52734181f-maas-api-openshift-ca\") pod \"e478b9aa-0913-49f0-82de-15d52734181f\" (UID: \"e478b9aa-0913-49f0-82de-15d52734181f\") " Nov 03 20:04:58.964815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:58.964800 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e478b9aa-0913-49f0-82de-15d52734181f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e478b9aa-0913-49f0-82de-15d52734181f-maas-api-xks-ca\") pod \"e478b9aa-0913-49f0-82de-15d52734181f\" (UID: \"e478b9aa-0913-49f0-82de-15d52734181f\") " Nov 03 20:04:58.964900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:58.964885 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e478b9aa-0913-49f0-82de-15d52734181f-kube-api-access-n5vbk\" (UniqueName: \"kubernetes.io/projected/e478b9aa-0913-49f0-82de-15d52734181f-kube-api-access-n5vbk\") pod \"e478b9aa-0913-49f0-82de-15d52734181f\" (UID: \"e478b9aa-0913-49f0-82de-15d52734181f\") " Nov 03 20:04:58.968642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:58.968598 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e478b9aa-0913-49f0-82de-15d52734181f-kube-api-access-n5vbk" pod "e478b9aa-0913-49f0-82de-15d52734181f" (UID: "e478b9aa-0913-49f0-82de-15d52734181f"). InnerVolumeSpecName "kube-api-access-n5vbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:58.969903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:58.969878 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1af6db2e-d6c9-47fd-a41e-c412988701d7-maas-api-openshift-ca" pod "1af6db2e-d6c9-47fd-a41e-c412988701d7" (UID: "1af6db2e-d6c9-47fd-a41e-c412988701d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:58.970015 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:58.969966 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e478b9aa-0913-49f0-82de-15d52734181f-maas-api-xks-ca" pod "e478b9aa-0913-49f0-82de-15d52734181f" (UID: "e478b9aa-0913-49f0-82de-15d52734181f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:04:58.982226 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:58.981988 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e478b9aa-0913-49f0-82de-15d52734181f-maas-api-openshift-ca" pod "e478b9aa-0913-49f0-82de-15d52734181f" (UID: "e478b9aa-0913-49f0-82de-15d52734181f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:59.002900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:59.002857 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af6db2e-d6c9-47fd-a41e-c412988701d7-kube-api-access-pdv7j" pod "1af6db2e-d6c9-47fd-a41e-c412988701d7" (UID: "1af6db2e-d6c9-47fd-a41e-c412988701d7"). InnerVolumeSpecName "kube-api-access-pdv7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:59.009253 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:59.009039 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1af6db2e-d6c9-47fd-a41e-c412988701d7-maas-api-xks-ca" pod "1af6db2e-d6c9-47fd-a41e-c412988701d7" (UID: "1af6db2e-d6c9-47fd-a41e-c412988701d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:04:59.067247 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:59.067212 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e478b9aa-0913-49f0-82de-15d52734181f-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:59.067247 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:59.067243 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5vbk\" (UniqueName: \"kubernetes.io/projected/e478b9aa-0913-49f0-82de-15d52734181f-kube-api-access-n5vbk\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:59.067453 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:59.067255 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af6db2e-d6c9-47fd-a41e-c412988701d7-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:59.067453 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:59.067266 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdv7j\" (UniqueName: \"kubernetes.io/projected/1af6db2e-d6c9-47fd-a41e-c412988701d7-kube-api-access-pdv7j\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:59.067453 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:59.067275 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af6db2e-d6c9-47fd-a41e-c412988701d7-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:59.067453 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:59.067283 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e478b9aa-0913-49f0-82de-15d52734181f-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:59.696863 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:59.696825 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" Nov 03 20:04:59.697335 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:59.697293 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9" event={"ID":"e478b9aa-0913-49f0-82de-15d52734181f","Type":"ContainerDied","Data":"c0f325ba8fbaee03b5e0c4cfd5bb06ae0e486083c7560c5f2e89614d2ea2c4ae"} Nov 03 20:04:59.697483 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:59.697358 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0f325ba8fbaee03b5e0c4cfd5bb06ae0e486083c7560c5f2e89614d2ea2c4ae" Nov 03 20:04:59.701624 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:59.701586 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" Nov 03 20:04:59.701771 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:59.701617 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm" event={"ID":"1af6db2e-d6c9-47fd-a41e-c412988701d7","Type":"ContainerDied","Data":"3e7081cc525eb4ecd106d429ffa4d987e3668acb8d0533c5dea1649f70927ab3"} Nov 03 20:04:59.701771 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:04:59.701679 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e7081cc525eb4ecd106d429ffa4d987e3668acb8d0533c5dea1649f70927ab3" Nov 03 20:05:03.191685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.191648 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs"] Nov 03 20:05:03.192462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.192443 5111 cpu_manager.go:414] "containerMap: removing container" podUID="1af6db2e-d6c9-47fd-a41e-c412988701d7" containerName="revoke-keys" Nov 03 20:05:03.192504 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.192465 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1af6db2e-d6c9-47fd-a41e-c412988701d7" containerName="revoke-keys" Nov 03 20:05:03.192504 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.192476 5111 cpu_manager.go:414] "containerMap: removing container" podUID="e478b9aa-0913-49f0-82de-15d52734181f" containerName="revoke-keys" Nov 03 20:05:03.192504 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.192481 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e478b9aa-0913-49f0-82de-15d52734181f" containerName="revoke-keys" Nov 03 20:05:03.192640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.192630 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="e478b9aa-0913-49f0-82de-15d52734181f" containerName="revoke-keys" Nov 03 20:05:03.192677 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.192654 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="1af6db2e-d6c9-47fd-a41e-c412988701d7" containerName="revoke-keys" Nov 03 20:05:03.196216 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.196195 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" Nov 03 20:05:03.203431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.203336 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:05:03.203431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.203366 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:05:03.203431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.203406 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:05:03.203743 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.203484 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:05:03.205829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.205802 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs"] Nov 03 20:05:03.350493 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.350450 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c86eb12-6efd-4389-9e92-7a5604ceea06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb465508-b9rqs\" (UID: \"9c86eb12-6efd-4389-9e92-7a5604ceea06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" Nov 03 20:05:03.350754 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.350628 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z8jj\" (UniqueName: \"kubernetes.io/projected/9c86eb12-6efd-4389-9e92-7a5604ceea06-kube-api-access-5z8jj\") pod \"maas-api-revoke-sub-eb465508-b9rqs\" (UID: \"9c86eb12-6efd-4389-9e92-7a5604ceea06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" Nov 03 20:05:03.350754 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.350712 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c86eb12-6efd-4389-9e92-7a5604ceea06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb465508-b9rqs\" (UID: \"9c86eb12-6efd-4389-9e92-7a5604ceea06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" Nov 03 20:05:03.452481 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.452393 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z8jj\" (UniqueName: \"kubernetes.io/projected/9c86eb12-6efd-4389-9e92-7a5604ceea06-kube-api-access-5z8jj\") pod \"maas-api-revoke-sub-eb465508-b9rqs\" (UID: \"9c86eb12-6efd-4389-9e92-7a5604ceea06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" Nov 03 20:05:03.452481 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.452479 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c86eb12-6efd-4389-9e92-7a5604ceea06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb465508-b9rqs\" (UID: \"9c86eb12-6efd-4389-9e92-7a5604ceea06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" Nov 03 20:05:03.452792 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.452694 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c86eb12-6efd-4389-9e92-7a5604ceea06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb465508-b9rqs\" (UID: \"9c86eb12-6efd-4389-9e92-7a5604ceea06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" Nov 03 20:05:03.453323 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.453299 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c86eb12-6efd-4389-9e92-7a5604ceea06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb465508-b9rqs\" (UID: \"9c86eb12-6efd-4389-9e92-7a5604ceea06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" Nov 03 20:05:03.455060 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.455040 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c86eb12-6efd-4389-9e92-7a5604ceea06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb465508-b9rqs\" (UID: \"9c86eb12-6efd-4389-9e92-7a5604ceea06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" Nov 03 20:05:03.460001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.459979 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z8jj\" (UniqueName: \"kubernetes.io/projected/9c86eb12-6efd-4389-9e92-7a5604ceea06-kube-api-access-5z8jj\") pod \"maas-api-revoke-sub-eb465508-b9rqs\" (UID: \"9c86eb12-6efd-4389-9e92-7a5604ceea06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" Nov 03 20:05:03.524633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.524599 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" Nov 03 20:05:03.815408 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:03.815359 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c86eb12_6efd_4389_9e92_7a5604ceea06.slice/crio-853a5a88b994b51832a0d2519222462b39ad1685d6b4e87c39b27b333759c7f5 WatchSource:0}: Error finding container 853a5a88b994b51832a0d2519222462b39ad1685d6b4e87c39b27b333759c7f5: Status 404 returned error can't find the container with id 853a5a88b994b51832a0d2519222462b39ad1685d6b4e87c39b27b333759c7f5 Nov 03 20:05:03.816591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:03.816561 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs"] Nov 03 20:05:04.158271 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.158235 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg"] Nov 03 20:05:04.164340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.164317 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" Nov 03 20:05:04.169502 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.169475 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg"] Nov 03 20:05:04.265450 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.265419 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxscn\" (UniqueName: \"kubernetes.io/projected/f8b664f5-92aa-4853-8335-25d0d6692a5c-kube-api-access-kxscn\") pod \"maas-api-revoke-sub-fd93879d-n4vhg\" (UID: \"f8b664f5-92aa-4853-8335-25d0d6692a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" Nov 03 20:05:04.265813 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.265559 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b664f5-92aa-4853-8335-25d0d6692a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd93879d-n4vhg\" (UID: \"f8b664f5-92aa-4853-8335-25d0d6692a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" Nov 03 20:05:04.265813 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.265657 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b664f5-92aa-4853-8335-25d0d6692a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd93879d-n4vhg\" (UID: \"f8b664f5-92aa-4853-8335-25d0d6692a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" Nov 03 20:05:04.367045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.367003 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b664f5-92aa-4853-8335-25d0d6692a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd93879d-n4vhg\" (UID: \"f8b664f5-92aa-4853-8335-25d0d6692a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" Nov 03 20:05:04.367250 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.367086 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b664f5-92aa-4853-8335-25d0d6692a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd93879d-n4vhg\" (UID: \"f8b664f5-92aa-4853-8335-25d0d6692a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" Nov 03 20:05:04.367250 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.367247 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxscn\" (UniqueName: \"kubernetes.io/projected/f8b664f5-92aa-4853-8335-25d0d6692a5c-kube-api-access-kxscn\") pod \"maas-api-revoke-sub-fd93879d-n4vhg\" (UID: \"f8b664f5-92aa-4853-8335-25d0d6692a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" Nov 03 20:05:04.367865 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.367839 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b664f5-92aa-4853-8335-25d0d6692a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd93879d-n4vhg\" (UID: \"f8b664f5-92aa-4853-8335-25d0d6692a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" Nov 03 20:05:04.369447 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.369423 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b664f5-92aa-4853-8335-25d0d6692a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd93879d-n4vhg\" (UID: \"f8b664f5-92aa-4853-8335-25d0d6692a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" Nov 03 20:05:04.375271 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.375249 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxscn\" (UniqueName: \"kubernetes.io/projected/f8b664f5-92aa-4853-8335-25d0d6692a5c-kube-api-access-kxscn\") pod \"maas-api-revoke-sub-fd93879d-n4vhg\" (UID: \"f8b664f5-92aa-4853-8335-25d0d6692a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" Nov 03 20:05:04.481020 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.480929 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" Nov 03 20:05:04.761059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.760928 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" event={"ID":"9c86eb12-6efd-4389-9e92-7a5604ceea06","Type":"ContainerStarted","Data":"7f74ecfc57e2923906aaaebe53d2991d079bd444f27a996004a2fd8eb2229645"} Nov 03 20:05:04.761059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.760999 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" event={"ID":"9c86eb12-6efd-4389-9e92-7a5604ceea06","Type":"ContainerStarted","Data":"853a5a88b994b51832a0d2519222462b39ad1685d6b4e87c39b27b333759c7f5"} Nov 03 20:05:04.780918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.780860 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" podStartSLOduration=1.7808429110000001 podStartE2EDuration="1.780842911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:04.777356417 +0000 UTC m=+1534.739734893" watchObservedRunningTime="2026-11-03 20:05:04.780842911 +0000 UTC m=+1534.743221398" Nov 03 20:05:04.805895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:04.805862 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg"] Nov 03 20:05:04.806145 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:04.806118 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b664f5_92aa_4853_8335_25d0d6692a5c.slice/crio-3198b38f62dfba9ace01e2277877adc03c66f1eec104a61caea83d6af45d5510 WatchSource:0}: Error finding container 3198b38f62dfba9ace01e2277877adc03c66f1eec104a61caea83d6af45d5510: Status 404 returned error can't find the container with id 3198b38f62dfba9ace01e2277877adc03c66f1eec104a61caea83d6af45d5510 Nov 03 20:05:05.059315 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:05:05.059280 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:05:05.774059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:05.773959 5111 generic.go:356] "Generic (PLEG): container finished" podID="9c86eb12-6efd-4389-9e92-7a5604ceea06" containerID="7f74ecfc57e2923906aaaebe53d2991d079bd444f27a996004a2fd8eb2229645" exitCode=0 Nov 03 20:05:05.774059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:05.774035 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" event={"ID":"9c86eb12-6efd-4389-9e92-7a5604ceea06","Type":"ContainerDied","Data":"7f74ecfc57e2923906aaaebe53d2991d079bd444f27a996004a2fd8eb2229645"} Nov 03 20:05:05.775915 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:05.775879 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" event={"ID":"f8b664f5-92aa-4853-8335-25d0d6692a5c","Type":"ContainerStarted","Data":"a48905df019bfbb67c2f7472bde88c8eb81b0bda3d10df3830c297059cbf1353"} Nov 03 20:05:05.776045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:05.775940 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" event={"ID":"f8b664f5-92aa-4853-8335-25d0d6692a5c","Type":"ContainerStarted","Data":"3198b38f62dfba9ace01e2277877adc03c66f1eec104a61caea83d6af45d5510"} Nov 03 20:05:05.807311 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:05.807251 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" podStartSLOduration=1.8072367809999998 podStartE2EDuration="1.807236781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:05.805007932 +0000 UTC m=+1535.767386371" watchObservedRunningTime="2026-11-03 20:05:05.807236781 +0000 UTC m=+1535.769615221" Nov 03 20:05:06.785763 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:06.785729 5111 generic.go:356] "Generic (PLEG): container finished" podID="f8b664f5-92aa-4853-8335-25d0d6692a5c" containerID="a48905df019bfbb67c2f7472bde88c8eb81b0bda3d10df3830c297059cbf1353" exitCode=0 Nov 03 20:05:06.786238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:06.785795 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" event={"ID":"f8b664f5-92aa-4853-8335-25d0d6692a5c","Type":"ContainerDied","Data":"a48905df019bfbb67c2f7472bde88c8eb81b0bda3d10df3830c297059cbf1353"} Nov 03 20:05:06.960659 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:06.960631 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" Nov 03 20:05:07.096906 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.096828 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm"] Nov 03 20:05:07.097555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.097523 5111 cpu_manager.go:414] "containerMap: removing container" podUID="9c86eb12-6efd-4389-9e92-7a5604ceea06" containerName="revoke-keys" Nov 03 20:05:07.097597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.097557 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c86eb12-6efd-4389-9e92-7a5604ceea06" containerName="revoke-keys" Nov 03 20:05:07.097681 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.097672 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c86eb12-6efd-4389-9e92-7a5604ceea06" containerName="revoke-keys" Nov 03 20:05:07.101459 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.101437 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.114876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.114850 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm"] Nov 03 20:05:07.127106 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.127074 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c86eb12-6efd-4389-9e92-7a5604ceea06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c86eb12-6efd-4389-9e92-7a5604ceea06-maas-api-openshift-ca\") pod \"9c86eb12-6efd-4389-9e92-7a5604ceea06\" (UID: \"9c86eb12-6efd-4389-9e92-7a5604ceea06\") " Nov 03 20:05:07.127276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.127179 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c86eb12-6efd-4389-9e92-7a5604ceea06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c86eb12-6efd-4389-9e92-7a5604ceea06-maas-api-xks-ca\") pod \"9c86eb12-6efd-4389-9e92-7a5604ceea06\" (UID: \"9c86eb12-6efd-4389-9e92-7a5604ceea06\") " Nov 03 20:05:07.127378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.127363 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c86eb12-6efd-4389-9e92-7a5604ceea06-kube-api-access-5z8jj\" (UniqueName: \"kubernetes.io/projected/9c86eb12-6efd-4389-9e92-7a5604ceea06-kube-api-access-5z8jj\") pod \"9c86eb12-6efd-4389-9e92-7a5604ceea06\" (UID: \"9c86eb12-6efd-4389-9e92-7a5604ceea06\") " Nov 03 20:05:07.127486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.127454 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c86eb12-6efd-4389-9e92-7a5604ceea06-maas-api-openshift-ca" pod "9c86eb12-6efd-4389-9e92-7a5604ceea06" (UID: "9c86eb12-6efd-4389-9e92-7a5604ceea06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:07.128901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.128137 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c86eb12-6efd-4389-9e92-7a5604ceea06-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:07.130166 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.130140 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c86eb12-6efd-4389-9e92-7a5604ceea06-maas-api-xks-ca" pod "9c86eb12-6efd-4389-9e92-7a5604ceea06" (UID: "9c86eb12-6efd-4389-9e92-7a5604ceea06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:07.130455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.130423 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c86eb12-6efd-4389-9e92-7a5604ceea06-kube-api-access-5z8jj" pod "9c86eb12-6efd-4389-9e92-7a5604ceea06" (UID: "9c86eb12-6efd-4389-9e92-7a5604ceea06"). InnerVolumeSpecName "kube-api-access-5z8jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:07.231280 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.231224 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.231280 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.231284 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.231640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.231437 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.231640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.231505 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.231640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.231571 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.231815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.231602 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.231815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.231787 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.234637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.231948 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v9cn\" (UniqueName: \"kubernetes.io/projected/2a40c5e8-4cd0-4e49-8a87-9b578902940b-kube-api-access-7v9cn\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.234637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.232099 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.234637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.232258 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z8jj\" (UniqueName: \"kubernetes.io/projected/9c86eb12-6efd-4389-9e92-7a5604ceea06-kube-api-access-5z8jj\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:07.234637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.232279 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c86eb12-6efd-4389-9e92-7a5604ceea06-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:07.333740 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.333695 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v9cn\" (UniqueName: \"kubernetes.io/projected/2a40c5e8-4cd0-4e49-8a87-9b578902940b-kube-api-access-7v9cn\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.333972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.333881 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.333972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.333932 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.333972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.333955 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.334158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.334001 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.334158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.334047 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.334158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.334075 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.334483 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.334453 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.334661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.334639 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.335058 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.335037 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.335324 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.335304 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.335360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.334457 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.335725 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.335703 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.335947 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.335926 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.336864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.336837 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.337899 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.337875 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.341824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.341802 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v9cn\" (UniqueName: \"kubernetes.io/projected/2a40c5e8-4cd0-4e49-8a87-9b578902940b-kube-api-access-7v9cn\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.343353 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.343330 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.417477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.417453 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:07.701618 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:07.701586 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a40c5e8_4cd0_4e49_8a87_9b578902940b.slice/crio-5d00d59947e6e31dea803b2e14be1e0f38399aa1f993beb1e54b45122fe387a5 WatchSource:0}: Error finding container 5d00d59947e6e31dea803b2e14be1e0f38399aa1f993beb1e54b45122fe387a5: Status 404 returned error can't find the container with id 5d00d59947e6e31dea803b2e14be1e0f38399aa1f993beb1e54b45122fe387a5 Nov 03 20:05:07.702489 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.702464 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm"] Nov 03 20:05:07.704329 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.704297 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:07.704440 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.704377 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:07.704440 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.704418 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:07.774735 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.774709 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l"] Nov 03 20:05:07.781365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.781340 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:07.807131 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.807101 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l"] Nov 03 20:05:07.817549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.814904 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" event={"ID":"2a40c5e8-4cd0-4e49-8a87-9b578902940b","Type":"ContainerStarted","Data":"6094075973e28c7300334d3d09882f60c70f021592cef901d963daa8bd92d782"} Nov 03 20:05:07.817549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.814963 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" event={"ID":"2a40c5e8-4cd0-4e49-8a87-9b578902940b","Type":"ContainerStarted","Data":"5d00d59947e6e31dea803b2e14be1e0f38399aa1f993beb1e54b45122fe387a5"} Nov 03 20:05:07.817549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.816664 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" Nov 03 20:05:07.817549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.816741 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs" event={"ID":"9c86eb12-6efd-4389-9e92-7a5604ceea06","Type":"ContainerDied","Data":"853a5a88b994b51832a0d2519222462b39ad1685d6b4e87c39b27b333759c7f5"} Nov 03 20:05:07.817549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.816792 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="853a5a88b994b51832a0d2519222462b39ad1685d6b4e87c39b27b333759c7f5" Nov 03 20:05:07.846745 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.846689 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" podStartSLOduration=0.846670049 podStartE2EDuration="846.670049ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:07.836010749 +0000 UTC m=+1537.798389189" watchObservedRunningTime="2026-11-03 20:05:07.846670049 +0000 UTC m=+1537.809048490" Nov 03 20:05:07.945817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.943967 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:07.945817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.944017 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:07.945817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.944052 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kktmn\" (UniqueName: \"kubernetes.io/projected/8f272466-fdc2-4521-9f68-782e0cb8da27-kube-api-access-kktmn\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:07.945817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.944148 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:07.945817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.944304 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:07.945817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.944357 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:07.945817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.944418 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:07.945817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.944457 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:07.945817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:07.944510 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f272466-fdc2-4521-9f68-782e0cb8da27-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.057606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.048401 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f272466-fdc2-4521-9f68-782e0cb8da27-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.057606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.048618 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.057606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.048651 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.057606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.048737 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kktmn\" (UniqueName: \"kubernetes.io/projected/8f272466-fdc2-4521-9f68-782e0cb8da27-kube-api-access-kktmn\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.057606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.048832 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.057606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.048944 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.057606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.048995 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.057606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.049058 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.057606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.049097 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.057606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.049703 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.057606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.050758 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f272466-fdc2-4521-9f68-782e0cb8da27-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.057606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.051072 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.057606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.051355 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.057606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.054910 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.058864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.058834 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.059838 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.059813 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.061089 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.061059 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kktmn\" (UniqueName: \"kubernetes.io/projected/8f272466-fdc2-4521-9f68-782e0cb8da27-kube-api-access-kktmn\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.062478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.062454 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" Nov 03 20:05:08.062669 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.062638 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.115347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.115304 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:08.150462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.150428 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8b664f5-92aa-4853-8335-25d0d6692a5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b664f5-92aa-4853-8335-25d0d6692a5c-maas-api-xks-ca\") pod \"f8b664f5-92aa-4853-8335-25d0d6692a5c\" (UID: \"f8b664f5-92aa-4853-8335-25d0d6692a5c\") " Nov 03 20:05:08.150647 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.150468 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b664f5-92aa-4853-8335-25d0d6692a5c-kube-api-access-kxscn\" (UniqueName: \"kubernetes.io/projected/f8b664f5-92aa-4853-8335-25d0d6692a5c-kube-api-access-kxscn\") pod \"f8b664f5-92aa-4853-8335-25d0d6692a5c\" (UID: \"f8b664f5-92aa-4853-8335-25d0d6692a5c\") " Nov 03 20:05:08.150647 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.150549 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b664f5-92aa-4853-8335-25d0d6692a5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b664f5-92aa-4853-8335-25d0d6692a5c-maas-api-openshift-ca\") pod \"f8b664f5-92aa-4853-8335-25d0d6692a5c\" (UID: \"f8b664f5-92aa-4853-8335-25d0d6692a5c\") " Nov 03 20:05:08.151356 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.151297 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b664f5-92aa-4853-8335-25d0d6692a5c-maas-api-openshift-ca" pod "f8b664f5-92aa-4853-8335-25d0d6692a5c" (UID: "f8b664f5-92aa-4853-8335-25d0d6692a5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:08.155184 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.155156 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8b664f5-92aa-4853-8335-25d0d6692a5c-maas-api-xks-ca" pod "f8b664f5-92aa-4853-8335-25d0d6692a5c" (UID: "f8b664f5-92aa-4853-8335-25d0d6692a5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:08.161497 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.161470 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b664f5-92aa-4853-8335-25d0d6692a5c-kube-api-access-kxscn" pod "f8b664f5-92aa-4853-8335-25d0d6692a5c" (UID: "f8b664f5-92aa-4853-8335-25d0d6692a5c"). InnerVolumeSpecName "kube-api-access-kxscn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:08.257712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.257679 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b664f5-92aa-4853-8335-25d0d6692a5c-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:08.257712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.257713 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxscn\" (UniqueName: \"kubernetes.io/projected/f8b664f5-92aa-4853-8335-25d0d6692a5c-kube-api-access-kxscn\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:08.257972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.257729 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b664f5-92aa-4853-8335-25d0d6692a5c-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:08.423157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.422402 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:08.430078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.430049 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l"] Nov 03 20:05:08.430208 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.430169 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:05:08.430350 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.430213 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" podUID="2a40c5e8-4cd0-4e49-8a87-9b578902940b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:05:08.433463 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:08.433431 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f272466_fdc2_4521_9f68_782e0cb8da27.slice/crio-58c215fb70ab1f5af040f45fe8edaaec4ee2dd2d590efed9b17cc60b2d50778a WatchSource:0}: Error finding container 58c215fb70ab1f5af040f45fe8edaaec4ee2dd2d590efed9b17cc60b2d50778a: Status 404 returned error can't find the container with id 58c215fb70ab1f5af040f45fe8edaaec4ee2dd2d590efed9b17cc60b2d50778a Nov 03 20:05:08.436429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.436396 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:08.436628 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.436465 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:08.436628 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.436496 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:08.617331 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.617288 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f"] Nov 03 20:05:08.618235 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.618216 5111 cpu_manager.go:414] "containerMap: removing container" podUID="f8b664f5-92aa-4853-8335-25d0d6692a5c" containerName="revoke-keys" Nov 03 20:05:08.618235 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.618235 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b664f5-92aa-4853-8335-25d0d6692a5c" containerName="revoke-keys" Nov 03 20:05:08.618378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.618374 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b664f5-92aa-4853-8335-25d0d6692a5c" containerName="revoke-keys" Nov 03 20:05:08.622591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.622540 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f" Nov 03 20:05:08.630591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.630564 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f"] Nov 03 20:05:08.693599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.690691 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a844b87-157c-4b4d-80f7-7b8bca7161f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00198b38-kcx4f\" (UID: \"1a844b87-157c-4b4d-80f7-7b8bca7161f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f" Nov 03 20:05:08.693599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.690735 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a844b87-157c-4b4d-80f7-7b8bca7161f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00198b38-kcx4f\" (UID: \"1a844b87-157c-4b4d-80f7-7b8bca7161f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f" Nov 03 20:05:08.693599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.690875 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spqhq\" (UniqueName: \"kubernetes.io/projected/1a844b87-157c-4b4d-80f7-7b8bca7161f1-kube-api-access-spqhq\") pod \"maas-api-revoke-sub-00198b38-kcx4f\" (UID: \"1a844b87-157c-4b4d-80f7-7b8bca7161f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f" Nov 03 20:05:08.792622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.792591 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spqhq\" (UniqueName: \"kubernetes.io/projected/1a844b87-157c-4b4d-80f7-7b8bca7161f1-kube-api-access-spqhq\") pod \"maas-api-revoke-sub-00198b38-kcx4f\" (UID: \"1a844b87-157c-4b4d-80f7-7b8bca7161f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f" Nov 03 20:05:08.792809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.792753 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a844b87-157c-4b4d-80f7-7b8bca7161f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00198b38-kcx4f\" (UID: \"1a844b87-157c-4b4d-80f7-7b8bca7161f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f" Nov 03 20:05:08.792809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.792783 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a844b87-157c-4b4d-80f7-7b8bca7161f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00198b38-kcx4f\" (UID: \"1a844b87-157c-4b4d-80f7-7b8bca7161f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f" Nov 03 20:05:08.793455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.793432 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a844b87-157c-4b4d-80f7-7b8bca7161f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00198b38-kcx4f\" (UID: \"1a844b87-157c-4b4d-80f7-7b8bca7161f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f" Nov 03 20:05:08.795279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.795260 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a844b87-157c-4b4d-80f7-7b8bca7161f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00198b38-kcx4f\" (UID: \"1a844b87-157c-4b4d-80f7-7b8bca7161f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f" Nov 03 20:05:08.801994 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.801969 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spqhq\" (UniqueName: \"kubernetes.io/projected/1a844b87-157c-4b4d-80f7-7b8bca7161f1-kube-api-access-spqhq\") pod \"maas-api-revoke-sub-00198b38-kcx4f\" (UID: \"1a844b87-157c-4b4d-80f7-7b8bca7161f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f" Nov 03 20:05:08.827762 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.827731 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" event={"ID":"8f272466-fdc2-4521-9f68-782e0cb8da27","Type":"ContainerStarted","Data":"3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39"} Nov 03 20:05:08.828095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.827793 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" event={"ID":"8f272466-fdc2-4521-9f68-782e0cb8da27","Type":"ContainerStarted","Data":"58c215fb70ab1f5af040f45fe8edaaec4ee2dd2d590efed9b17cc60b2d50778a"} Nov 03 20:05:08.829369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.829352 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" Nov 03 20:05:08.829447 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.829380 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg" event={"ID":"f8b664f5-92aa-4853-8335-25d0d6692a5c","Type":"ContainerDied","Data":"3198b38f62dfba9ace01e2277877adc03c66f1eec104a61caea83d6af45d5510"} Nov 03 20:05:08.829447 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.829416 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3198b38f62dfba9ace01e2277877adc03c66f1eec104a61caea83d6af45d5510" Nov 03 20:05:08.870968 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.870906 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" podStartSLOduration=1.870888405 podStartE2EDuration="1.870888405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:08.869964679 +0000 UTC m=+1538.832343119" watchObservedRunningTime="2026-11-03 20:05:08.870888405 +0000 UTC m=+1538.833266845" Nov 03 20:05:08.994060 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:08.993985 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f" Nov 03 20:05:09.117164 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:09.117131 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:09.148743 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:09.143686 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:05:09.148743 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:09.143744 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" podUID="8f272466-fdc2-4521-9f68-782e0cb8da27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:05:09.417889 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:09.417812 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:05:09.418146 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:09.417924 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" podUID="2a40c5e8-4cd0-4e49-8a87-9b578902940b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:05:09.531970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:09.531937 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f"] Nov 03 20:05:09.838761 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:09.838677 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f" event={"ID":"1a844b87-157c-4b4d-80f7-7b8bca7161f1","Type":"ContainerStarted","Data":"7dedaddb228d9b6aa052f2d8fe016f09968a789d9d4756cf353bfc8ae4e8007c"} Nov 03 20:05:10.123560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:10.115957 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:05:10.123560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:10.116022 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" podUID="8f272466-fdc2-4521-9f68-782e0cb8da27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:05:10.418983 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:10.418514 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:05:10.418983 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:10.418609 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" podUID="2a40c5e8-4cd0-4e49-8a87-9b578902940b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:05:10.902593 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:10.902397 5111 generic.go:356] "Generic (PLEG): container finished" podID="1a844b87-157c-4b4d-80f7-7b8bca7161f1" containerID="29cb735bef6d13bdc62970ce8d59f6818b290990a9dc2d0627927af23573350a" exitCode=0 Nov 03 20:05:10.902593 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:10.902438 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f" event={"ID":"1a844b87-157c-4b4d-80f7-7b8bca7161f1","Type":"ContainerDied","Data":"29cb735bef6d13bdc62970ce8d59f6818b290990a9dc2d0627927af23573350a"} Nov 03 20:05:11.021780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.021677 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b"] Nov 03 20:05:11.026657 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.026636 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" Nov 03 20:05:11.034405 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.034378 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b"] Nov 03 20:05:11.116208 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.116162 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:05:11.116409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.116250 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" podUID="8f272466-fdc2-4521-9f68-782e0cb8da27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:05:11.213559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.213046 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be1befb4-1127-459d-8a13-f4d71450c0a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2944544-x4s4b\" (UID: \"be1befb4-1127-459d-8a13-f4d71450c0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" Nov 03 20:05:11.213559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.213215 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlfcg\" (UniqueName: \"kubernetes.io/projected/be1befb4-1127-459d-8a13-f4d71450c0a8-kube-api-access-mlfcg\") pod \"maas-api-revoke-sub-a2944544-x4s4b\" (UID: \"be1befb4-1127-459d-8a13-f4d71450c0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" Nov 03 20:05:11.213559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.213469 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be1befb4-1127-459d-8a13-f4d71450c0a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2944544-x4s4b\" (UID: \"be1befb4-1127-459d-8a13-f4d71450c0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" Nov 03 20:05:11.315527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.315435 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be1befb4-1127-459d-8a13-f4d71450c0a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2944544-x4s4b\" (UID: \"be1befb4-1127-459d-8a13-f4d71450c0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" Nov 03 20:05:11.315729 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.315610 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlfcg\" (UniqueName: \"kubernetes.io/projected/be1befb4-1127-459d-8a13-f4d71450c0a8-kube-api-access-mlfcg\") pod \"maas-api-revoke-sub-a2944544-x4s4b\" (UID: \"be1befb4-1127-459d-8a13-f4d71450c0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" Nov 03 20:05:11.315729 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.315636 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be1befb4-1127-459d-8a13-f4d71450c0a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2944544-x4s4b\" (UID: \"be1befb4-1127-459d-8a13-f4d71450c0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" Nov 03 20:05:11.316152 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.316128 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be1befb4-1127-459d-8a13-f4d71450c0a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2944544-x4s4b\" (UID: \"be1befb4-1127-459d-8a13-f4d71450c0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" Nov 03 20:05:11.318739 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.318716 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be1befb4-1127-459d-8a13-f4d71450c0a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2944544-x4s4b\" (UID: \"be1befb4-1127-459d-8a13-f4d71450c0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" Nov 03 20:05:11.325267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.325248 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlfcg\" (UniqueName: \"kubernetes.io/projected/be1befb4-1127-459d-8a13-f4d71450c0a8-kube-api-access-mlfcg\") pod \"maas-api-revoke-sub-a2944544-x4s4b\" (UID: \"be1befb4-1127-459d-8a13-f4d71450c0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" Nov 03 20:05:11.416019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.415987 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" Nov 03 20:05:11.418250 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.418222 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:05:11.418394 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.418363 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" podUID="2a40c5e8-4cd0-4e49-8a87-9b578902940b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:05:11.803730 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.803679 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b"] Nov 03 20:05:11.806850 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:11.806819 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe1befb4_1127_459d_8a13_f4d71450c0a8.slice/crio-671295b2899a2f9c24cde053d03b6ccbc47c1b199435182a53028a1da77e1ac8 WatchSource:0}: Error finding container 671295b2899a2f9c24cde053d03b6ccbc47c1b199435182a53028a1da77e1ac8: Status 404 returned error can't find the container with id 671295b2899a2f9c24cde053d03b6ccbc47c1b199435182a53028a1da77e1ac8 Nov 03 20:05:11.921299 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:11.921227 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" event={"ID":"be1befb4-1127-459d-8a13-f4d71450c0a8","Type":"ContainerStarted","Data":"671295b2899a2f9c24cde053d03b6ccbc47c1b199435182a53028a1da77e1ac8"} Nov 03 20:05:12.116718 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.116682 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:05:12.117001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.116970 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" podUID="8f272466-fdc2-4521-9f68-782e0cb8da27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:05:12.408412 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.408389 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f" Nov 03 20:05:12.419323 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.419293 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:05:12.419449 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.419338 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" podUID="2a40c5e8-4cd0-4e49-8a87-9b578902940b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:05:12.442920 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.442879 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a844b87-157c-4b4d-80f7-7b8bca7161f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a844b87-157c-4b4d-80f7-7b8bca7161f1-maas-api-xks-ca\") pod \"1a844b87-157c-4b4d-80f7-7b8bca7161f1\" (UID: \"1a844b87-157c-4b4d-80f7-7b8bca7161f1\") " Nov 03 20:05:12.443153 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.442972 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a844b87-157c-4b4d-80f7-7b8bca7161f1-kube-api-access-spqhq\" (UniqueName: \"kubernetes.io/projected/1a844b87-157c-4b4d-80f7-7b8bca7161f1-kube-api-access-spqhq\") pod \"1a844b87-157c-4b4d-80f7-7b8bca7161f1\" (UID: \"1a844b87-157c-4b4d-80f7-7b8bca7161f1\") " Nov 03 20:05:12.443153 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.443082 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a844b87-157c-4b4d-80f7-7b8bca7161f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a844b87-157c-4b4d-80f7-7b8bca7161f1-maas-api-openshift-ca\") pod \"1a844b87-157c-4b4d-80f7-7b8bca7161f1\" (UID: \"1a844b87-157c-4b4d-80f7-7b8bca7161f1\") " Nov 03 20:05:12.444225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.444196 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a844b87-157c-4b4d-80f7-7b8bca7161f1-maas-api-openshift-ca" pod "1a844b87-157c-4b4d-80f7-7b8bca7161f1" (UID: "1a844b87-157c-4b4d-80f7-7b8bca7161f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:12.509607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.509564 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a844b87-157c-4b4d-80f7-7b8bca7161f1-maas-api-xks-ca" pod "1a844b87-157c-4b4d-80f7-7b8bca7161f1" (UID: "1a844b87-157c-4b4d-80f7-7b8bca7161f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:12.511285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.511251 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a844b87-157c-4b4d-80f7-7b8bca7161f1-kube-api-access-spqhq" pod "1a844b87-157c-4b4d-80f7-7b8bca7161f1" (UID: "1a844b87-157c-4b4d-80f7-7b8bca7161f1"). InnerVolumeSpecName "kube-api-access-spqhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:12.574560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.545150 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a844b87-157c-4b4d-80f7-7b8bca7161f1-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:12.608551 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.606052 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spqhq\" (UniqueName: \"kubernetes.io/projected/1a844b87-157c-4b4d-80f7-7b8bca7161f1-kube-api-access-spqhq\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:12.608551 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.606077 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a844b87-157c-4b4d-80f7-7b8bca7161f1-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:12.934846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.934818 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f" Nov 03 20:05:12.935327 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.934860 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f" event={"ID":"1a844b87-157c-4b4d-80f7-7b8bca7161f1","Type":"ContainerDied","Data":"7dedaddb228d9b6aa052f2d8fe016f09968a789d9d4756cf353bfc8ae4e8007c"} Nov 03 20:05:12.935327 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.934920 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dedaddb228d9b6aa052f2d8fe016f09968a789d9d4756cf353bfc8ae4e8007c" Nov 03 20:05:12.938026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:12.937991 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" event={"ID":"be1befb4-1127-459d-8a13-f4d71450c0a8","Type":"ContainerStarted","Data":"61ed8d8180cbe69d50b6cbdaeba88e98ec17106f1e7e9f390d81c855e73282bd"} Nov 03 20:05:13.036381 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:13.036309 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" podStartSLOduration=3.036287273 podStartE2EDuration="3.036287273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:12.974860315 +0000 UTC m=+1542.937238754" watchObservedRunningTime="2026-11-03 20:05:13.036287273 +0000 UTC m=+1542.998665713" Nov 03 20:05:13.116214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:13.116170 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:05:13.116406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:13.116232 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" podUID="8f272466-fdc2-4521-9f68-782e0cb8da27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:05:13.418508 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:13.418476 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:05:13.418703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:13.418548 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" podUID="2a40c5e8-4cd0-4e49-8a87-9b578902940b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:05:13.994067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:13.991917 5111 generic.go:356] "Generic (PLEG): container finished" podID="be1befb4-1127-459d-8a13-f4d71450c0a8" containerID="61ed8d8180cbe69d50b6cbdaeba88e98ec17106f1e7e9f390d81c855e73282bd" exitCode=0 Nov 03 20:05:13.994067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:13.992201 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" event={"ID":"be1befb4-1127-459d-8a13-f4d71450c0a8","Type":"ContainerDied","Data":"61ed8d8180cbe69d50b6cbdaeba88e98ec17106f1e7e9f390d81c855e73282bd"} Nov 03 20:05:14.116302 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:14.116268 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:05:14.116504 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:14.116329 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" podUID="8f272466-fdc2-4521-9f68-782e0cb8da27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:05:14.418177 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:14.418145 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:05:14.418337 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:14.418198 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" podUID="2a40c5e8-4cd0-4e49-8a87-9b578902940b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:05:15.116206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:15.116165 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:05:15.116801 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:15.116231 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" podUID="8f272466-fdc2-4521-9f68-782e0cb8da27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:05:15.235388 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:15.235359 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" Nov 03 20:05:15.362674 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:15.362563 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be1befb4-1127-459d-8a13-f4d71450c0a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be1befb4-1127-459d-8a13-f4d71450c0a8-maas-api-xks-ca\") pod \"be1befb4-1127-459d-8a13-f4d71450c0a8\" (UID: \"be1befb4-1127-459d-8a13-f4d71450c0a8\") " Nov 03 20:05:15.362674 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:15.362632 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1befb4-1127-459d-8a13-f4d71450c0a8-kube-api-access-mlfcg\" (UniqueName: \"kubernetes.io/projected/be1befb4-1127-459d-8a13-f4d71450c0a8-kube-api-access-mlfcg\") pod \"be1befb4-1127-459d-8a13-f4d71450c0a8\" (UID: \"be1befb4-1127-459d-8a13-f4d71450c0a8\") " Nov 03 20:05:15.362929 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:15.362847 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be1befb4-1127-459d-8a13-f4d71450c0a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be1befb4-1127-459d-8a13-f4d71450c0a8-maas-api-openshift-ca\") pod \"be1befb4-1127-459d-8a13-f4d71450c0a8\" (UID: \"be1befb4-1127-459d-8a13-f4d71450c0a8\") " Nov 03 20:05:15.363696 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:15.363663 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be1befb4-1127-459d-8a13-f4d71450c0a8-maas-api-openshift-ca" pod "be1befb4-1127-459d-8a13-f4d71450c0a8" (UID: "be1befb4-1127-459d-8a13-f4d71450c0a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:15.363879 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:15.363853 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be1befb4-1127-459d-8a13-f4d71450c0a8-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:15.365874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:15.365843 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1befb4-1127-459d-8a13-f4d71450c0a8-kube-api-access-mlfcg" pod "be1befb4-1127-459d-8a13-f4d71450c0a8" (UID: "be1befb4-1127-459d-8a13-f4d71450c0a8"). InnerVolumeSpecName "kube-api-access-mlfcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:15.366044 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:15.366024 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be1befb4-1127-459d-8a13-f4d71450c0a8-maas-api-xks-ca" pod "be1befb4-1127-459d-8a13-f4d71450c0a8" (UID: "be1befb4-1127-459d-8a13-f4d71450c0a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:15.465768 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:15.465718 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be1befb4-1127-459d-8a13-f4d71450c0a8-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:15.465768 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:15.465765 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlfcg\" (UniqueName: \"kubernetes.io/projected/be1befb4-1127-459d-8a13-f4d71450c0a8-kube-api-access-mlfcg\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:15.567896 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:15.567861 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:15.569647 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:15.569621 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:15.571304 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:15.571279 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:05:16.016436 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:16.016394 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" Nov 03 20:05:16.017794 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:16.017748 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b" event={"ID":"be1befb4-1127-459d-8a13-f4d71450c0a8","Type":"ContainerDied","Data":"671295b2899a2f9c24cde053d03b6ccbc47c1b199435182a53028a1da77e1ac8"} Nov 03 20:05:16.017959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:16.017829 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="671295b2899a2f9c24cde053d03b6ccbc47c1b199435182a53028a1da77e1ac8" Nov 03 20:05:16.219597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:16.219562 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:16.220054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:16.220031 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:16.221019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:16.220997 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:05:20.470031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.469994 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4"] Nov 03 20:05:20.470773 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.470754 5111 cpu_manager.go:414] "containerMap: removing container" podUID="be1befb4-1127-459d-8a13-f4d71450c0a8" containerName="revoke-keys" Nov 03 20:05:20.470832 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.470773 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1befb4-1127-459d-8a13-f4d71450c0a8" containerName="revoke-keys" Nov 03 20:05:20.470832 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.470799 5111 cpu_manager.go:414] "containerMap: removing container" podUID="1a844b87-157c-4b4d-80f7-7b8bca7161f1" containerName="revoke-keys" Nov 03 20:05:20.470832 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.470805 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a844b87-157c-4b4d-80f7-7b8bca7161f1" containerName="revoke-keys" Nov 03 20:05:20.470946 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.470936 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1befb4-1127-459d-8a13-f4d71450c0a8" containerName="revoke-keys" Nov 03 20:05:20.470982 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.470951 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a844b87-157c-4b4d-80f7-7b8bca7161f1" containerName="revoke-keys" Nov 03 20:05:20.475127 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.475104 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" Nov 03 20:05:20.483042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.482989 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:05:20.483042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.482991 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:05:20.484557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.484153 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:05:20.484557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.484219 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:05:20.486030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.486003 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4"] Nov 03 20:05:20.561206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.561169 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kxh7\" (UniqueName: \"kubernetes.io/projected/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-kube-api-access-4kxh7\") pod \"maas-api-revoke-sub-edf9fa38-86cd4\" (UID: \"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" Nov 03 20:05:20.561627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.561601 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edf9fa38-86cd4\" (UID: \"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" Nov 03 20:05:20.561756 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.561645 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edf9fa38-86cd4\" (UID: \"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" Nov 03 20:05:20.663156 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.663121 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kxh7\" (UniqueName: \"kubernetes.io/projected/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-kube-api-access-4kxh7\") pod \"maas-api-revoke-sub-edf9fa38-86cd4\" (UID: \"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" Nov 03 20:05:20.663364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.663272 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edf9fa38-86cd4\" (UID: \"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" Nov 03 20:05:20.663364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.663298 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edf9fa38-86cd4\" (UID: \"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" Nov 03 20:05:20.664213 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.664182 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edf9fa38-86cd4\" (UID: \"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" Nov 03 20:05:20.665781 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.665753 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edf9fa38-86cd4\" (UID: \"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" Nov 03 20:05:20.671218 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.671191 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kxh7\" (UniqueName: \"kubernetes.io/projected/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-kube-api-access-4kxh7\") pod \"maas-api-revoke-sub-edf9fa38-86cd4\" (UID: \"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" Nov 03 20:05:20.807306 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:20.807219 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" Nov 03 20:05:21.115947 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:21.115905 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4"] Nov 03 20:05:21.124182 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:21.124143 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e82679a_ae4d_463d_9ca7_a7a4cd0ccd65.slice/crio-2e15c78e84e0b03a09a3fa42c619f048905bcbd7ce027fd27bff97a78c7b159c WatchSource:0}: Error finding container 2e15c78e84e0b03a09a3fa42c619f048905bcbd7ce027fd27bff97a78c7b159c: Status 404 returned error can't find the container with id 2e15c78e84e0b03a09a3fa42c619f048905bcbd7ce027fd27bff97a78c7b159c Nov 03 20:05:22.109559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:22.104428 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" event={"ID":"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65","Type":"ContainerStarted","Data":"de0f16263e507e8489f073c0996472ef4f5e13327d2144bd5c01050d9387383a"} Nov 03 20:05:22.109559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:22.104561 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" event={"ID":"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65","Type":"ContainerStarted","Data":"2e15c78e84e0b03a09a3fa42c619f048905bcbd7ce027fd27bff97a78c7b159c"} Nov 03 20:05:22.169957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:22.164937 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" podStartSLOduration=2.164913359 podStartE2EDuration="2.164913359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:22.126286285 +0000 UTC m=+1552.088664725" watchObservedRunningTime="2026-11-03 20:05:22.164913359 +0000 UTC m=+1552.127291804" Nov 03 20:05:22.949585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:22.898732 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:23.380658 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:23.357053 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:23.393571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:23.390698 5111 generic.go:356] "Generic (PLEG): container finished" podID="5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65" containerID="de0f16263e507e8489f073c0996472ef4f5e13327d2144bd5c01050d9387383a" exitCode=0 Nov 03 20:05:23.393571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:23.390949 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" event={"ID":"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65","Type":"ContainerDied","Data":"de0f16263e507e8489f073c0996472ef4f5e13327d2144bd5c01050d9387383a"} Nov 03 20:05:23.417078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:23.417029 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:23.419012 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:23.418958 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:23.511197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:23.511149 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:23.556499 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:23.556432 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:23.557647 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:23.557607 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:23.594566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:23.577697 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 20:05:23.685619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:23.682803 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 20:05:23.685939 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:23.685895 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:23.729561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:23.728989 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:23.778686 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:23.770207 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:24.361726 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.361675 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:24.362685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.362655 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:05:24.422713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.422662 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:05:24.439560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.438498 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:05:24.476560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.440601 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:05:24.534421 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.534384 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:05:24.598585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.534461 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:05:24.718609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.706647 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:05:24.840560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.593319 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:05:24.879558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.637409 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:24.903154 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.637460 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Nov 03 20:05:24.971295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.637840 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 20:05:25.011589 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.687407 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:25.165558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.687445 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 20:05:25.224614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.593280 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:05:25.240557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.800013 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 20:05:25.240557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.827743 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:25.240557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:24.876938 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:05:25.240557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:25.225199 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:05:25.240557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:25.225242 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:05:25.240557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:25.225277 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:05:25.240557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:25.225311 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:05:25.240557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:25.225342 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:05:25.240557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:25.225375 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:05:25.396007 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:25.378631 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 20:05:25.439571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:25.425477 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:25.480294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:25.465636 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:25.530766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:25.530727 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded" Nov 03 20:05:25.660559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:25.624977 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:25.717743 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:25.717695 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:26.164556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:26.113382 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:05:26.164556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:26.151671 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:26.165783 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:26.165729 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:26.212558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:26.188293 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 20:05:26.249974 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:26.249924 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:26.317799 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:26.317752 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:05:26.318299 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:26.318267 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 20:05:26.385798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:26.382622 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:26.476477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:26.476411 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:26.669666 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:26.669617 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:26.719009 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:26.718915 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:26.806559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:26.737643 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:05:27.195555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.195482 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:27.209658 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.209571 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:05:27.236943 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.236885 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:27.237140 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.237077 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:27.239938 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.239874 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:05:27.253840 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.253796 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:27.298615 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.279841 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:05:27.298615 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.283278 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:05:27.318199 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.312228 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:05:27.318199 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.314991 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:05:27.320145 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.320113 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:05:27.320461 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.320432 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:05:27.339225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.339198 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:05:27.347442 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.347260 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:05:27.359407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.356115 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:05:27.366660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.363479 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:05:27.366660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.365070 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" Nov 03 20:05:27.371070 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.371040 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn"] Nov 03 20:05:27.372405 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.372380 5111 cpu_manager.go:414] "containerMap: removing container" podUID="5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65" containerName="revoke-keys" Nov 03 20:05:27.372601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.372584 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65" containerName="revoke-keys" Nov 03 20:05:27.372948 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.372929 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65" containerName="revoke-keys" Nov 03 20:05:27.377670 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.377595 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" Nov 03 20:05:27.385663 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.385635 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn"] Nov 03 20:05:27.464268 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.464214 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" Nov 03 20:05:27.465160 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.464489 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4" event={"ID":"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65","Type":"ContainerDied","Data":"2e15c78e84e0b03a09a3fa42c619f048905bcbd7ce027fd27bff97a78c7b159c"} Nov 03 20:05:27.465160 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.464599 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e15c78e84e0b03a09a3fa42c619f048905bcbd7ce027fd27bff97a78c7b159c" Nov 03 20:05:27.545318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.545280 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-kube-api-access-4kxh7\" (UniqueName: \"kubernetes.io/projected/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-kube-api-access-4kxh7\") pod \"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65\" (UID: \"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65\") " Nov 03 20:05:27.546019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.545989 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-maas-api-xks-ca\") pod \"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65\" (UID: \"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65\") " Nov 03 20:05:27.546166 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.546120 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-maas-api-openshift-ca\") pod \"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65\" (UID: \"5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65\") " Nov 03 20:05:27.546685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.546636 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f8fw\" (UniqueName: \"kubernetes.io/projected/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-kube-api-access-8f8fw\") pod \"maas-api-revoke-sub-ff3be028-bz5kn\" (UID: \"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" Nov 03 20:05:27.546986 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.546965 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff3be028-bz5kn\" (UID: \"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" Nov 03 20:05:27.547703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.547678 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff3be028-bz5kn\" (UID: \"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" Nov 03 20:05:27.547822 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.547785 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-maas-api-openshift-ca" pod "5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65" (UID: "5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:27.555612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.552781 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-maas-api-xks-ca" pod "5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65" (UID: "5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:27.555612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.554731 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-kube-api-access-4kxh7" pod "5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65" (UID: "5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65"). InnerVolumeSpecName "kube-api-access-4kxh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:27.648926 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.648886 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f8fw\" (UniqueName: \"kubernetes.io/projected/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-kube-api-access-8f8fw\") pod \"maas-api-revoke-sub-ff3be028-bz5kn\" (UID: \"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" Nov 03 20:05:27.649122 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.648951 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff3be028-bz5kn\" (UID: \"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" Nov 03 20:05:27.649441 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.649416 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff3be028-bz5kn\" (UID: \"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" Nov 03 20:05:27.652164 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.650399 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:27.652164 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.651266 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:27.652164 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.651899 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kxh7\" (UniqueName: \"kubernetes.io/projected/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65-kube-api-access-4kxh7\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:27.652164 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.652097 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff3be028-bz5kn\" (UID: \"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" Nov 03 20:05:27.653926 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.653901 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff3be028-bz5kn\" (UID: \"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" Nov 03 20:05:27.668398 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.668360 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f8fw\" (UniqueName: \"kubernetes.io/projected/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-kube-api-access-8f8fw\") pod \"maas-api-revoke-sub-ff3be028-bz5kn\" (UID: \"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" Nov 03 20:05:27.728333 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:27.728294 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" Nov 03 20:05:28.054914 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.054882 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn"] Nov 03 20:05:28.055584 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:28.055514 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17da1d9a_61f2_4f7e_a4d0_0904b72e6fd0.slice/crio-1f3a88c2b59196c2477c8e0442a58f5a7a36a262fef0d57e69974175847a0c5c WatchSource:0}: Error finding container 1f3a88c2b59196c2477c8e0442a58f5a7a36a262fef0d57e69974175847a0c5c: Status 404 returned error can't find the container with id 1f3a88c2b59196c2477c8e0442a58f5a7a36a262fef0d57e69974175847a0c5c Nov 03 20:05:28.109214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.109184 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947"] Nov 03 20:05:28.115806 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.115781 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" Nov 03 20:05:28.128927 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.128898 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947"] Nov 03 20:05:28.272841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.269387 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74f492cf-s7947\" (UID: \"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" Nov 03 20:05:28.272841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.269723 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbpwm\" (UniqueName: \"kubernetes.io/projected/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-kube-api-access-bbpwm\") pod \"maas-api-revoke-sub-74f492cf-s7947\" (UID: \"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" Nov 03 20:05:28.272841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.269783 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74f492cf-s7947\" (UID: \"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" Nov 03 20:05:28.371754 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.371713 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbpwm\" (UniqueName: \"kubernetes.io/projected/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-kube-api-access-bbpwm\") pod \"maas-api-revoke-sub-74f492cf-s7947\" (UID: \"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" Nov 03 20:05:28.371940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.371769 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74f492cf-s7947\" (UID: \"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" Nov 03 20:05:28.371940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.371914 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74f492cf-s7947\" (UID: \"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" Nov 03 20:05:28.372689 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.372659 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74f492cf-s7947\" (UID: \"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" Nov 03 20:05:28.374693 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.374526 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74f492cf-s7947\" (UID: \"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" Nov 03 20:05:28.384455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.384434 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbpwm\" (UniqueName: \"kubernetes.io/projected/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-kube-api-access-bbpwm\") pod \"maas-api-revoke-sub-74f492cf-s7947\" (UID: \"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" Nov 03 20:05:28.446675 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.446646 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" Nov 03 20:05:28.489236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.489199 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" event={"ID":"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0","Type":"ContainerStarted","Data":"da29161987bb6c06b2efe57549218ff9cf49daf1c922c4c9c912e3e65638191d"} Nov 03 20:05:28.489371 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.489295 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" event={"ID":"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0","Type":"ContainerStarted","Data":"1f3a88c2b59196c2477c8e0442a58f5a7a36a262fef0d57e69974175847a0c5c"} Nov 03 20:05:28.511475 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.511414 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" podStartSLOduration=3.511395224 podStartE2EDuration="3.511395224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:28.50642792 +0000 UTC m=+1558.468806359" watchObservedRunningTime="2026-11-03 20:05:28.511395224 +0000 UTC m=+1558.473773664" Nov 03 20:05:28.811476 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.811385 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947"] Nov 03 20:05:28.829504 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:28.829470 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2961e7a6_1f7c_4ad3_b7e5_35c661b4af6b.slice/crio-8434bfbe83ee22ff23283b9fcf4e730de4fb1965236aa0a171fed1f9272139d0 WatchSource:0}: Error finding container 8434bfbe83ee22ff23283b9fcf4e730de4fb1965236aa0a171fed1f9272139d0: Status 404 returned error can't find the container with id 8434bfbe83ee22ff23283b9fcf4e730de4fb1965236aa0a171fed1f9272139d0 Nov 03 20:05:28.928059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.928020 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh"] Nov 03 20:05:28.933842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.933821 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" Nov 03 20:05:28.946433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:28.946399 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh"] Nov 03 20:05:29.087397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:29.087320 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6eb5149-k6xmh\" (UID: \"596f1b95-fc67-4fea-bbb5-48c915d1fd2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" Nov 03 20:05:29.087574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:29.087470 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgrs8\" (UniqueName: \"kubernetes.io/projected/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-kube-api-access-rgrs8\") pod \"maas-api-revoke-sub-c6eb5149-k6xmh\" (UID: \"596f1b95-fc67-4fea-bbb5-48c915d1fd2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" Nov 03 20:05:29.087625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:29.087609 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6eb5149-k6xmh\" (UID: \"596f1b95-fc67-4fea-bbb5-48c915d1fd2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" Nov 03 20:05:29.188742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:29.188712 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6eb5149-k6xmh\" (UID: \"596f1b95-fc67-4fea-bbb5-48c915d1fd2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" Nov 03 20:05:29.188891 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:29.188802 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgrs8\" (UniqueName: \"kubernetes.io/projected/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-kube-api-access-rgrs8\") pod \"maas-api-revoke-sub-c6eb5149-k6xmh\" (UID: \"596f1b95-fc67-4fea-bbb5-48c915d1fd2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" Nov 03 20:05:29.188891 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:29.188884 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6eb5149-k6xmh\" (UID: \"596f1b95-fc67-4fea-bbb5-48c915d1fd2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" Nov 03 20:05:29.189362 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:29.189341 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6eb5149-k6xmh\" (UID: \"596f1b95-fc67-4fea-bbb5-48c915d1fd2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" Nov 03 20:05:29.191197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:29.191175 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6eb5149-k6xmh\" (UID: \"596f1b95-fc67-4fea-bbb5-48c915d1fd2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" Nov 03 20:05:29.197518 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:29.197496 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgrs8\" (UniqueName: \"kubernetes.io/projected/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-kube-api-access-rgrs8\") pod \"maas-api-revoke-sub-c6eb5149-k6xmh\" (UID: \"596f1b95-fc67-4fea-bbb5-48c915d1fd2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" Nov 03 20:05:29.254287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:29.254206 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" Nov 03 20:05:29.505686 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:29.505324 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" event={"ID":"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b","Type":"ContainerStarted","Data":"05f70d46188789c08cc80e9a7aa997a4dc418d58604ded3acd5ce0e886093c26"} Nov 03 20:05:29.505686 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:29.505391 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" event={"ID":"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b","Type":"ContainerStarted","Data":"8434bfbe83ee22ff23283b9fcf4e730de4fb1965236aa0a171fed1f9272139d0"} Nov 03 20:05:29.507867 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:29.507825 5111 generic.go:356] "Generic (PLEG): container finished" podID="17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0" containerID="da29161987bb6c06b2efe57549218ff9cf49daf1c922c4c9c912e3e65638191d" exitCode=0 Nov 03 20:05:29.508166 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:29.507899 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" event={"ID":"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0","Type":"ContainerDied","Data":"da29161987bb6c06b2efe57549218ff9cf49daf1c922c4c9c912e3e65638191d"} Nov 03 20:05:29.537409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:29.530806 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" podStartSLOduration=1.530788722 podStartE2EDuration="1.530788722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:29.528953385 +0000 UTC m=+1559.491331841" watchObservedRunningTime="2026-11-03 20:05:29.530788722 +0000 UTC m=+1559.493167164" Nov 03 20:05:29.552569 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:29.552521 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596f1b95_fc67_4fea_bbb5_48c915d1fd2e.slice/crio-dfdb152a76dfba5fe20581f89c4844b575211026376a087e7897fb57ebfb1577 WatchSource:0}: Error finding container dfdb152a76dfba5fe20581f89c4844b575211026376a087e7897fb57ebfb1577: Status 404 returned error can't find the container with id dfdb152a76dfba5fe20581f89c4844b575211026376a087e7897fb57ebfb1577 Nov 03 20:05:29.558396 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:29.558371 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh"] Nov 03 20:05:30.520035 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:30.519988 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" event={"ID":"596f1b95-fc67-4fea-bbb5-48c915d1fd2e","Type":"ContainerStarted","Data":"e206111691da4522d2e1846fac0fe8a287a367c0c4b652ee67bcbf3bfca7f36c"} Nov 03 20:05:30.520551 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:30.520090 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" event={"ID":"596f1b95-fc67-4fea-bbb5-48c915d1fd2e","Type":"ContainerStarted","Data":"dfdb152a76dfba5fe20581f89c4844b575211026376a087e7897fb57ebfb1577"} Nov 03 20:05:30.521835 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:30.521804 5111 generic.go:356] "Generic (PLEG): container finished" podID="2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b" containerID="05f70d46188789c08cc80e9a7aa997a4dc418d58604ded3acd5ce0e886093c26" exitCode=0 Nov 03 20:05:30.521953 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:30.521861 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" event={"ID":"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b","Type":"ContainerDied","Data":"05f70d46188789c08cc80e9a7aa997a4dc418d58604ded3acd5ce0e886093c26"} Nov 03 20:05:30.543459 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:30.543409 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" podStartSLOduration=2.54339399 podStartE2EDuration="2.54339399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:30.540482393 +0000 UTC m=+1560.502860833" watchObservedRunningTime="2026-11-03 20:05:30.54339399 +0000 UTC m=+1560.505772438" Nov 03 20:05:30.740287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:30.740262 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" Nov 03 20:05:30.820630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:30.820547 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-kube-api-access-8f8fw\" (UniqueName: \"kubernetes.io/projected/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-kube-api-access-8f8fw\") pod \"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0\" (UID: \"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0\") " Nov 03 20:05:30.820630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:30.820611 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-maas-api-xks-ca\") pod \"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0\" (UID: \"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0\") " Nov 03 20:05:30.820842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:30.820656 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-maas-api-openshift-ca\") pod \"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0\" (UID: \"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0\") " Nov 03 20:05:30.821098 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:30.821073 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-maas-api-openshift-ca" pod "17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0" (UID: "17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:30.821400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:30.821378 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:30.822885 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:30.822862 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-maas-api-xks-ca" pod "17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0" (UID: "17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:30.823069 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:30.823052 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-kube-api-access-8f8fw" pod "17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0" (UID: "17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0"). InnerVolumeSpecName "kube-api-access-8f8fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:30.922580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:30.922548 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f8fw\" (UniqueName: \"kubernetes.io/projected/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-kube-api-access-8f8fw\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:30.922580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:30.922577 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:31.532838 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:31.532806 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" Nov 03 20:05:31.533300 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:31.532840 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn" event={"ID":"17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0","Type":"ContainerDied","Data":"1f3a88c2b59196c2477c8e0442a58f5a7a36a262fef0d57e69974175847a0c5c"} Nov 03 20:05:31.533300 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:31.532890 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f3a88c2b59196c2477c8e0442a58f5a7a36a262fef0d57e69974175847a0c5c" Nov 03 20:05:31.534891 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:31.534867 5111 generic.go:356] "Generic (PLEG): container finished" podID="596f1b95-fc67-4fea-bbb5-48c915d1fd2e" containerID="e206111691da4522d2e1846fac0fe8a287a367c0c4b652ee67bcbf3bfca7f36c" exitCode=0 Nov 03 20:05:31.535019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:31.534936 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" event={"ID":"596f1b95-fc67-4fea-bbb5-48c915d1fd2e","Type":"ContainerDied","Data":"e206111691da4522d2e1846fac0fe8a287a367c0c4b652ee67bcbf3bfca7f36c"} Nov 03 20:05:31.736188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:31.736159 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" Nov 03 20:05:31.835390 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:31.835311 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-maas-api-xks-ca\") pod \"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b\" (UID: \"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b\") " Nov 03 20:05:31.835526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:31.835485 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-maas-api-openshift-ca\") pod \"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b\" (UID: \"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b\") " Nov 03 20:05:31.835655 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:31.835627 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-kube-api-access-bbpwm\" (UniqueName: \"kubernetes.io/projected/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-kube-api-access-bbpwm\") pod \"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b\" (UID: \"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b\") " Nov 03 20:05:31.835957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:31.835882 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-maas-api-openshift-ca" pod "2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b" (UID: "2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:31.836337 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:31.836317 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:31.837830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:31.837799 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-kube-api-access-bbpwm" pod "2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b" (UID: "2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b"). InnerVolumeSpecName "kube-api-access-bbpwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:31.837962 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:31.837938 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-maas-api-xks-ca" pod "2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b" (UID: "2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:31.937870 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:31.937839 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbpwm\" (UniqueName: \"kubernetes.io/projected/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-kube-api-access-bbpwm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:31.937870 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:31.937870 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:32.547506 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.547429 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" Nov 03 20:05:32.547506 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.547461 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947" event={"ID":"2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b","Type":"ContainerDied","Data":"8434bfbe83ee22ff23283b9fcf4e730de4fb1965236aa0a171fed1f9272139d0"} Nov 03 20:05:32.548023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.547514 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8434bfbe83ee22ff23283b9fcf4e730de4fb1965236aa0a171fed1f9272139d0" Nov 03 20:05:32.685302 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.685261 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd"] Nov 03 20:05:32.686467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.686440 5111 cpu_manager.go:414] "containerMap: removing container" podUID="17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0" containerName="revoke-keys" Nov 03 20:05:32.686805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.686766 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0" containerName="revoke-keys" Nov 03 20:05:32.687003 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.686968 5111 cpu_manager.go:414] "containerMap: removing container" podUID="2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b" containerName="revoke-keys" Nov 03 20:05:32.687003 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.686984 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b" containerName="revoke-keys" Nov 03 20:05:32.687289 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.687258 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0" containerName="revoke-keys" Nov 03 20:05:32.687289 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.687279 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b" containerName="revoke-keys" Nov 03 20:05:32.692159 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.692138 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd" Nov 03 20:05:32.695332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.695303 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd"] Nov 03 20:05:32.758084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.758057 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" Nov 03 20:05:32.766211 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.766178 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2548d1ba-2355-4d85-8e21-2938cefbbce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03cfbe52-98qzd\" (UID: \"2548d1ba-2355-4d85-8e21-2938cefbbce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd" Nov 03 20:05:32.766454 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.766432 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s48t4\" (UniqueName: \"kubernetes.io/projected/2548d1ba-2355-4d85-8e21-2938cefbbce4-kube-api-access-s48t4\") pod \"maas-api-revoke-sub-03cfbe52-98qzd\" (UID: \"2548d1ba-2355-4d85-8e21-2938cefbbce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd" Nov 03 20:05:32.766585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.766567 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2548d1ba-2355-4d85-8e21-2938cefbbce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03cfbe52-98qzd\" (UID: \"2548d1ba-2355-4d85-8e21-2938cefbbce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd" Nov 03 20:05:32.868394 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.868233 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-maas-api-xks-ca\") pod \"596f1b95-fc67-4fea-bbb5-48c915d1fd2e\" (UID: \"596f1b95-fc67-4fea-bbb5-48c915d1fd2e\") " Nov 03 20:05:32.868799 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.868774 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-kube-api-access-rgrs8\" (UniqueName: \"kubernetes.io/projected/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-kube-api-access-rgrs8\") pod \"596f1b95-fc67-4fea-bbb5-48c915d1fd2e\" (UID: \"596f1b95-fc67-4fea-bbb5-48c915d1fd2e\") " Nov 03 20:05:32.869091 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.869068 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-maas-api-openshift-ca\") pod \"596f1b95-fc67-4fea-bbb5-48c915d1fd2e\" (UID: \"596f1b95-fc67-4fea-bbb5-48c915d1fd2e\") " Nov 03 20:05:32.870319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.870163 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2548d1ba-2355-4d85-8e21-2938cefbbce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03cfbe52-98qzd\" (UID: \"2548d1ba-2355-4d85-8e21-2938cefbbce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd" Nov 03 20:05:32.870708 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.870666 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2548d1ba-2355-4d85-8e21-2938cefbbce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03cfbe52-98qzd\" (UID: \"2548d1ba-2355-4d85-8e21-2938cefbbce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd" Nov 03 20:05:32.871066 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.871043 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s48t4\" (UniqueName: \"kubernetes.io/projected/2548d1ba-2355-4d85-8e21-2938cefbbce4-kube-api-access-s48t4\") pod \"maas-api-revoke-sub-03cfbe52-98qzd\" (UID: \"2548d1ba-2355-4d85-8e21-2938cefbbce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd" Nov 03 20:05:32.871771 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.871731 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-maas-api-openshift-ca" pod "596f1b95-fc67-4fea-bbb5-48c915d1fd2e" (UID: "596f1b95-fc67-4fea-bbb5-48c915d1fd2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:32.872842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.872295 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2548d1ba-2355-4d85-8e21-2938cefbbce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03cfbe52-98qzd\" (UID: \"2548d1ba-2355-4d85-8e21-2938cefbbce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd" Nov 03 20:05:32.873884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.873847 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-kube-api-access-rgrs8" pod "596f1b95-fc67-4fea-bbb5-48c915d1fd2e" (UID: "596f1b95-fc67-4fea-bbb5-48c915d1fd2e"). InnerVolumeSpecName "kube-api-access-rgrs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:32.875125 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.875098 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-maas-api-xks-ca" pod "596f1b95-fc67-4fea-bbb5-48c915d1fd2e" (UID: "596f1b95-fc67-4fea-bbb5-48c915d1fd2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:32.875224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.875177 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2548d1ba-2355-4d85-8e21-2938cefbbce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03cfbe52-98qzd\" (UID: \"2548d1ba-2355-4d85-8e21-2938cefbbce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd" Nov 03 20:05:32.881337 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.881301 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s48t4\" (UniqueName: \"kubernetes.io/projected/2548d1ba-2355-4d85-8e21-2938cefbbce4-kube-api-access-s48t4\") pod \"maas-api-revoke-sub-03cfbe52-98qzd\" (UID: \"2548d1ba-2355-4d85-8e21-2938cefbbce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd" Nov 03 20:05:32.973328 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.973286 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:32.973328 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.973331 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgrs8\" (UniqueName: \"kubernetes.io/projected/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-kube-api-access-rgrs8\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:32.973614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:32.973349 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596f1b95-fc67-4fea-bbb5-48c915d1fd2e-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:33.056918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:33.056884 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd" Nov 03 20:05:33.351793 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:33.351759 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2548d1ba_2355_4d85_8e21_2938cefbbce4.slice/crio-d9638421d2bb6314ecc0e166ca5a0bd6a1759199fbd548b9efb019a63417fa24 WatchSource:0}: Error finding container d9638421d2bb6314ecc0e166ca5a0bd6a1759199fbd548b9efb019a63417fa24: Status 404 returned error can't find the container with id d9638421d2bb6314ecc0e166ca5a0bd6a1759199fbd548b9efb019a63417fa24 Nov 03 20:05:33.353281 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:33.353250 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd"] Nov 03 20:05:33.557732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:33.557697 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd" event={"ID":"2548d1ba-2355-4d85-8e21-2938cefbbce4","Type":"ContainerStarted","Data":"d9638421d2bb6314ecc0e166ca5a0bd6a1759199fbd548b9efb019a63417fa24"} Nov 03 20:05:33.559460 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:33.559416 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" Nov 03 20:05:33.559460 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:33.559437 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh" event={"ID":"596f1b95-fc67-4fea-bbb5-48c915d1fd2e","Type":"ContainerDied","Data":"dfdb152a76dfba5fe20581f89c4844b575211026376a087e7897fb57ebfb1577"} Nov 03 20:05:33.559662 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:33.559482 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfdb152a76dfba5fe20581f89c4844b575211026376a087e7897fb57ebfb1577" Nov 03 20:05:34.570931 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:34.570898 5111 generic.go:356] "Generic (PLEG): container finished" podID="2548d1ba-2355-4d85-8e21-2938cefbbce4" containerID="e175e427733c88e790141b5c4bae5e02ef236d33772f2ac2ba895fb3c9b0ed09" exitCode=0 Nov 03 20:05:34.571340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:34.570986 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd" event={"ID":"2548d1ba-2355-4d85-8e21-2938cefbbce4","Type":"ContainerDied","Data":"e175e427733c88e790141b5c4bae5e02ef236d33772f2ac2ba895fb3c9b0ed09"} Nov 03 20:05:35.057677 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:05:35.057639 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:05:35.770901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:35.770875 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd" Nov 03 20:05:35.815748 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:35.815716 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2548d1ba-2355-4d85-8e21-2938cefbbce4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2548d1ba-2355-4d85-8e21-2938cefbbce4-maas-api-xks-ca\") pod \"2548d1ba-2355-4d85-8e21-2938cefbbce4\" (UID: \"2548d1ba-2355-4d85-8e21-2938cefbbce4\") " Nov 03 20:05:35.815996 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:35.815976 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2548d1ba-2355-4d85-8e21-2938cefbbce4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2548d1ba-2355-4d85-8e21-2938cefbbce4-maas-api-openshift-ca\") pod \"2548d1ba-2355-4d85-8e21-2938cefbbce4\" (UID: \"2548d1ba-2355-4d85-8e21-2938cefbbce4\") " Nov 03 20:05:35.816094 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:35.816078 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2548d1ba-2355-4d85-8e21-2938cefbbce4-kube-api-access-s48t4\" (UniqueName: \"kubernetes.io/projected/2548d1ba-2355-4d85-8e21-2938cefbbce4-kube-api-access-s48t4\") pod \"2548d1ba-2355-4d85-8e21-2938cefbbce4\" (UID: \"2548d1ba-2355-4d85-8e21-2938cefbbce4\") " Nov 03 20:05:35.816347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:35.816321 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2548d1ba-2355-4d85-8e21-2938cefbbce4-maas-api-openshift-ca" pod "2548d1ba-2355-4d85-8e21-2938cefbbce4" (UID: "2548d1ba-2355-4d85-8e21-2938cefbbce4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:35.816745 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:35.816719 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2548d1ba-2355-4d85-8e21-2938cefbbce4-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:35.818418 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:35.818390 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2548d1ba-2355-4d85-8e21-2938cefbbce4-kube-api-access-s48t4" pod "2548d1ba-2355-4d85-8e21-2938cefbbce4" (UID: "2548d1ba-2355-4d85-8e21-2938cefbbce4"). InnerVolumeSpecName "kube-api-access-s48t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:35.818826 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:35.818805 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2548d1ba-2355-4d85-8e21-2938cefbbce4-maas-api-xks-ca" pod "2548d1ba-2355-4d85-8e21-2938cefbbce4" (UID: "2548d1ba-2355-4d85-8e21-2938cefbbce4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:35.918602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:35.918560 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2548d1ba-2355-4d85-8e21-2938cefbbce4-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:35.918750 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:35.918607 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s48t4\" (UniqueName: \"kubernetes.io/projected/2548d1ba-2355-4d85-8e21-2938cefbbce4-kube-api-access-s48t4\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.592191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:36.592117 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd" Nov 03 20:05:36.592191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:36.592153 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd" event={"ID":"2548d1ba-2355-4d85-8e21-2938cefbbce4","Type":"ContainerDied","Data":"d9638421d2bb6314ecc0e166ca5a0bd6a1759199fbd548b9efb019a63417fa24"} Nov 03 20:05:36.592360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:36.592202 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9638421d2bb6314ecc0e166ca5a0bd6a1759199fbd548b9efb019a63417fa24" Nov 03 20:05:37.347456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.347418 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86"] Nov 03 20:05:37.349019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.348990 5111 cpu_manager.go:414] "containerMap: removing container" podUID="596f1b95-fc67-4fea-bbb5-48c915d1fd2e" containerName="revoke-keys" Nov 03 20:05:37.349019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.349018 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596f1b95-fc67-4fea-bbb5-48c915d1fd2e" containerName="revoke-keys" Nov 03 20:05:37.349167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.349075 5111 cpu_manager.go:414] "containerMap: removing container" podUID="2548d1ba-2355-4d85-8e21-2938cefbbce4" containerName="revoke-keys" Nov 03 20:05:37.349167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.349082 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2548d1ba-2355-4d85-8e21-2938cefbbce4" containerName="revoke-keys" Nov 03 20:05:37.349238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.349226 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="2548d1ba-2355-4d85-8e21-2938cefbbce4" containerName="revoke-keys" Nov 03 20:05:37.349273 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.349243 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="596f1b95-fc67-4fea-bbb5-48c915d1fd2e" containerName="revoke-keys" Nov 03 20:05:37.353328 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.353307 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" Nov 03 20:05:37.359488 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.359461 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:05:37.359657 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.359586 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:05:37.364759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.364737 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:05:37.364852 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.364799 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:05:37.371873 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.371826 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86"] Nov 03 20:05:37.449694 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.449665 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25afd63c-h7p86\" (UID: \"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" Nov 03 20:05:37.449838 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.449702 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25afd63c-h7p86\" (UID: \"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" Nov 03 20:05:37.449838 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.449792 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gnl5\" (UniqueName: \"kubernetes.io/projected/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-kube-api-access-4gnl5\") pod \"maas-api-revoke-sub-25afd63c-h7p86\" (UID: \"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" Nov 03 20:05:37.551357 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.551324 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25afd63c-h7p86\" (UID: \"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" Nov 03 20:05:37.551520 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.551363 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25afd63c-h7p86\" (UID: \"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" Nov 03 20:05:37.551520 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.551427 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gnl5\" (UniqueName: \"kubernetes.io/projected/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-kube-api-access-4gnl5\") pod \"maas-api-revoke-sub-25afd63c-h7p86\" (UID: \"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" Nov 03 20:05:37.552015 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.551991 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25afd63c-h7p86\" (UID: \"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" Nov 03 20:05:37.553776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.553753 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25afd63c-h7p86\" (UID: \"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" Nov 03 20:05:37.585726 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.585700 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gnl5\" (UniqueName: \"kubernetes.io/projected/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-kube-api-access-4gnl5\") pod \"maas-api-revoke-sub-25afd63c-h7p86\" (UID: \"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" Nov 03 20:05:37.674645 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.674602 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" Nov 03 20:05:37.993019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:37.992980 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86"] Nov 03 20:05:38.613790 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:38.613684 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" event={"ID":"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54","Type":"ContainerStarted","Data":"8ba3921ac17d1ca1247fe90a397f0e35a75e6b80075699cde6f0205e7af15c77"} Nov 03 20:05:38.613790 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:38.613754 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" event={"ID":"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54","Type":"ContainerStarted","Data":"187d4926b8361816824ea1991dc5bbfeded0ec9ea84727e0c474e5445b99b5e5"} Nov 03 20:05:38.633843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:38.633795 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" podStartSLOduration=1.633782138 podStartE2EDuration="1.633782138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:38.631263227 +0000 UTC m=+1568.593641667" watchObservedRunningTime="2026-11-03 20:05:38.633782138 +0000 UTC m=+1568.596160577" Nov 03 20:05:39.625707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:39.625668 5111 generic.go:356] "Generic (PLEG): container finished" podID="15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54" containerID="8ba3921ac17d1ca1247fe90a397f0e35a75e6b80075699cde6f0205e7af15c77" exitCode=0 Nov 03 20:05:39.626166 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:39.625754 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" event={"ID":"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54","Type":"ContainerDied","Data":"8ba3921ac17d1ca1247fe90a397f0e35a75e6b80075699cde6f0205e7af15c77"} Nov 03 20:05:40.837094 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:40.837063 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" Nov 03 20:05:41.005911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.005824 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-maas-api-xks-ca\") pod \"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54\" (UID: \"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54\") " Nov 03 20:05:41.005911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.005888 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-kube-api-access-4gnl5\" (UniqueName: \"kubernetes.io/projected/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-kube-api-access-4gnl5\") pod \"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54\" (UID: \"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54\") " Nov 03 20:05:41.006099 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.005995 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-maas-api-openshift-ca\") pod \"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54\" (UID: \"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54\") " Nov 03 20:05:41.006371 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.006342 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-maas-api-openshift-ca" pod "15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54" (UID: "15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:41.006663 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.006645 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:41.008394 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.008371 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-maas-api-xks-ca" pod "15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54" (UID: "15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:41.008984 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.008964 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-kube-api-access-4gnl5" pod "15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54" (UID: "15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54"). InnerVolumeSpecName "kube-api-access-4gnl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:41.101807 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.099810 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d"] Nov 03 20:05:41.101807 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.100987 5111 cpu_manager.go:414] "containerMap: removing container" podUID="15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54" containerName="revoke-keys" Nov 03 20:05:41.101807 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.101024 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54" containerName="revoke-keys" Nov 03 20:05:41.101807 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.101296 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54" containerName="revoke-keys" Nov 03 20:05:41.113484 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.112714 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" Nov 03 20:05:41.133473 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.131397 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d"] Nov 03 20:05:41.141640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.139840 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76d5576-0a53-4653-bb66-c8e640e6f819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15a654e1-f4g8d\" (UID: \"d76d5576-0a53-4653-bb66-c8e640e6f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" Nov 03 20:05:41.141640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.140134 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76d5576-0a53-4653-bb66-c8e640e6f819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15a654e1-f4g8d\" (UID: \"d76d5576-0a53-4653-bb66-c8e640e6f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" Nov 03 20:05:41.141640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.140212 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnc5d\" (UniqueName: \"kubernetes.io/projected/d76d5576-0a53-4653-bb66-c8e640e6f819-kube-api-access-wnc5d\") pod \"maas-api-revoke-sub-15a654e1-f4g8d\" (UID: \"d76d5576-0a53-4653-bb66-c8e640e6f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" Nov 03 20:05:41.141640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.140284 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:41.141640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.140314 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gnl5\" (UniqueName: \"kubernetes.io/projected/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54-kube-api-access-4gnl5\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:41.241960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.241924 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76d5576-0a53-4653-bb66-c8e640e6f819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15a654e1-f4g8d\" (UID: \"d76d5576-0a53-4653-bb66-c8e640e6f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" Nov 03 20:05:41.242140 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.242026 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnc5d\" (UniqueName: \"kubernetes.io/projected/d76d5576-0a53-4653-bb66-c8e640e6f819-kube-api-access-wnc5d\") pod \"maas-api-revoke-sub-15a654e1-f4g8d\" (UID: \"d76d5576-0a53-4653-bb66-c8e640e6f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" Nov 03 20:05:41.242140 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.242121 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76d5576-0a53-4653-bb66-c8e640e6f819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15a654e1-f4g8d\" (UID: \"d76d5576-0a53-4653-bb66-c8e640e6f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" Nov 03 20:05:41.243138 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.243107 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76d5576-0a53-4653-bb66-c8e640e6f819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15a654e1-f4g8d\" (UID: \"d76d5576-0a53-4653-bb66-c8e640e6f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" Nov 03 20:05:41.244480 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.244452 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76d5576-0a53-4653-bb66-c8e640e6f819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15a654e1-f4g8d\" (UID: \"d76d5576-0a53-4653-bb66-c8e640e6f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" Nov 03 20:05:41.251162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.251139 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnc5d\" (UniqueName: \"kubernetes.io/projected/d76d5576-0a53-4653-bb66-c8e640e6f819-kube-api-access-wnc5d\") pod \"maas-api-revoke-sub-15a654e1-f4g8d\" (UID: \"d76d5576-0a53-4653-bb66-c8e640e6f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" Nov 03 20:05:41.456205 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.456167 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" Nov 03 20:05:41.658111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.658064 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" event={"ID":"15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54","Type":"ContainerDied","Data":"187d4926b8361816824ea1991dc5bbfeded0ec9ea84727e0c474e5445b99b5e5"} Nov 03 20:05:41.658111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.658106 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86" Nov 03 20:05:41.658364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.658141 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="187d4926b8361816824ea1991dc5bbfeded0ec9ea84727e0c474e5445b99b5e5" Nov 03 20:05:41.748564 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:41.748502 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76d5576_0a53_4653_bb66_c8e640e6f819.slice/crio-a22c799645d20e8e40135fc978f5e3e4b153e1bcb3f881eade0246af567f8675 WatchSource:0}: Error finding container a22c799645d20e8e40135fc978f5e3e4b153e1bcb3f881eade0246af567f8675: Status 404 returned error can't find the container with id a22c799645d20e8e40135fc978f5e3e4b153e1bcb3f881eade0246af567f8675 Nov 03 20:05:41.751713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:41.751685 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d"] Nov 03 20:05:42.198568 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.198519 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv"] Nov 03 20:05:42.203269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.203247 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" Nov 03 20:05:42.212941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.212916 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv"] Nov 03 20:05:42.253978 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.253944 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll"] Nov 03 20:05:42.259055 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.259030 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" Nov 03 20:05:42.266427 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.266401 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjxjz\" (UniqueName: \"kubernetes.io/projected/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-kube-api-access-rjxjz\") pod \"maas-api-revoke-sub-08d7663c-r5wdv\" (UID: \"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" Nov 03 20:05:42.266787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.266762 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d7663c-r5wdv\" (UID: \"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" Nov 03 20:05:42.267255 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.267228 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d7663c-r5wdv\" (UID: \"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" Nov 03 20:05:42.268874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.268852 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll"] Nov 03 20:05:42.368563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.368515 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d7663c-r5wdv\" (UID: \"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" Nov 03 20:05:42.368723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.368645 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p6vt\" (UniqueName: \"kubernetes.io/projected/11c72ddb-a97a-43b1-b26a-7814772a7d4c-kube-api-access-7p6vt\") pod \"maas-api-revoke-sub-b6274d31-fv6ll\" (UID: \"11c72ddb-a97a-43b1-b26a-7814772a7d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" Nov 03 20:05:42.368723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.368668 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjxjz\" (UniqueName: \"kubernetes.io/projected/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-kube-api-access-rjxjz\") pod \"maas-api-revoke-sub-08d7663c-r5wdv\" (UID: \"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" Nov 03 20:05:42.368723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.368687 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c72ddb-a97a-43b1-b26a-7814772a7d4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6274d31-fv6ll\" (UID: \"11c72ddb-a97a-43b1-b26a-7814772a7d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" Nov 03 20:05:42.368723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.368709 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d7663c-r5wdv\" (UID: \"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" Nov 03 20:05:42.368986 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.368965 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c72ddb-a97a-43b1-b26a-7814772a7d4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6274d31-fv6ll\" (UID: \"11c72ddb-a97a-43b1-b26a-7814772a7d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" Nov 03 20:05:42.369288 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.369265 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d7663c-r5wdv\" (UID: \"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" Nov 03 20:05:42.370965 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.370946 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d7663c-r5wdv\" (UID: \"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" Nov 03 20:05:42.381936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.381908 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjxjz\" (UniqueName: \"kubernetes.io/projected/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-kube-api-access-rjxjz\") pod \"maas-api-revoke-sub-08d7663c-r5wdv\" (UID: \"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" Nov 03 20:05:42.470777 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.470679 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p6vt\" (UniqueName: \"kubernetes.io/projected/11c72ddb-a97a-43b1-b26a-7814772a7d4c-kube-api-access-7p6vt\") pod \"maas-api-revoke-sub-b6274d31-fv6ll\" (UID: \"11c72ddb-a97a-43b1-b26a-7814772a7d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" Nov 03 20:05:42.470777 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.470744 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c72ddb-a97a-43b1-b26a-7814772a7d4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6274d31-fv6ll\" (UID: \"11c72ddb-a97a-43b1-b26a-7814772a7d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" Nov 03 20:05:42.471010 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.470883 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c72ddb-a97a-43b1-b26a-7814772a7d4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6274d31-fv6ll\" (UID: \"11c72ddb-a97a-43b1-b26a-7814772a7d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" Nov 03 20:05:42.471545 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.471508 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c72ddb-a97a-43b1-b26a-7814772a7d4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6274d31-fv6ll\" (UID: \"11c72ddb-a97a-43b1-b26a-7814772a7d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" Nov 03 20:05:42.473963 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.473927 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c72ddb-a97a-43b1-b26a-7814772a7d4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6274d31-fv6ll\" (UID: \"11c72ddb-a97a-43b1-b26a-7814772a7d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" Nov 03 20:05:42.478432 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.478410 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p6vt\" (UniqueName: \"kubernetes.io/projected/11c72ddb-a97a-43b1-b26a-7814772a7d4c-kube-api-access-7p6vt\") pod \"maas-api-revoke-sub-b6274d31-fv6ll\" (UID: \"11c72ddb-a97a-43b1-b26a-7814772a7d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" Nov 03 20:05:42.535469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.535391 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" Nov 03 20:05:42.588024 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.587986 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" Nov 03 20:05:42.674157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.674119 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" event={"ID":"d76d5576-0a53-4653-bb66-c8e640e6f819","Type":"ContainerStarted","Data":"a09a49d56097c64ebeac3a64efe70b0c840bd3694d59ed034d55dddf6ee0da3a"} Nov 03 20:05:42.674267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.674186 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" event={"ID":"d76d5576-0a53-4653-bb66-c8e640e6f819","Type":"ContainerStarted","Data":"a22c799645d20e8e40135fc978f5e3e4b153e1bcb3f881eade0246af567f8675"} Nov 03 20:05:42.702352 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.702300 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" podStartSLOduration=1.7022851239999999 podStartE2EDuration="1.702285124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:42.694820859 +0000 UTC m=+1572.657199299" watchObservedRunningTime="2026-11-03 20:05:42.702285124 +0000 UTC m=+1572.664663563" Nov 03 20:05:42.831889 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:42.831854 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee7b794_4f05_4e4f_8b1d_3a6994f32a7f.slice/crio-112a8ea29f81fce65f43e31876deabb30ec5ba5f29659b5a3dd9971b6b366803 WatchSource:0}: Error finding container 112a8ea29f81fce65f43e31876deabb30ec5ba5f29659b5a3dd9971b6b366803: Status 404 returned error can't find the container with id 112a8ea29f81fce65f43e31876deabb30ec5ba5f29659b5a3dd9971b6b366803 Nov 03 20:05:42.832991 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.832961 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv"] Nov 03 20:05:42.903427 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:42.903400 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll"] Nov 03 20:05:42.956611 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:42.956584 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c72ddb_a97a_43b1_b26a_7814772a7d4c.slice/crio-5297f89f53fc8105dce06c3308aafa455482b650d711fb700f8379dab6c946a6 WatchSource:0}: Error finding container 5297f89f53fc8105dce06c3308aafa455482b650d711fb700f8379dab6c946a6: Status 404 returned error can't find the container with id 5297f89f53fc8105dce06c3308aafa455482b650d711fb700f8379dab6c946a6 Nov 03 20:05:43.685001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:43.684970 5111 generic.go:356] "Generic (PLEG): container finished" podID="d76d5576-0a53-4653-bb66-c8e640e6f819" containerID="a09a49d56097c64ebeac3a64efe70b0c840bd3694d59ed034d55dddf6ee0da3a" exitCode=0 Nov 03 20:05:43.685441 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:43.685051 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" event={"ID":"d76d5576-0a53-4653-bb66-c8e640e6f819","Type":"ContainerDied","Data":"a09a49d56097c64ebeac3a64efe70b0c840bd3694d59ed034d55dddf6ee0da3a"} Nov 03 20:05:43.686877 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:43.686849 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" event={"ID":"11c72ddb-a97a-43b1-b26a-7814772a7d4c","Type":"ContainerStarted","Data":"51eaeb3fafc316ca0e9ed18103022e7e8016b21572ce144a39d3a27320ff5d15"} Nov 03 20:05:43.687008 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:43.686898 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" event={"ID":"11c72ddb-a97a-43b1-b26a-7814772a7d4c","Type":"ContainerStarted","Data":"5297f89f53fc8105dce06c3308aafa455482b650d711fb700f8379dab6c946a6"} Nov 03 20:05:43.688733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:43.688708 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" event={"ID":"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f","Type":"ContainerStarted","Data":"af50fef6d72621b9b1cde5712f2677a5ebf06fca89a753025c4954e9032ecde9"} Nov 03 20:05:43.688841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:43.688753 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" event={"ID":"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f","Type":"ContainerStarted","Data":"112a8ea29f81fce65f43e31876deabb30ec5ba5f29659b5a3dd9971b6b366803"} Nov 03 20:05:43.722156 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:43.722110 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" podStartSLOduration=1.72209476 podStartE2EDuration="1.72209476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:43.719591113 +0000 UTC m=+1573.681969554" watchObservedRunningTime="2026-11-03 20:05:43.72209476 +0000 UTC m=+1573.684473201" Nov 03 20:05:43.737585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:43.737520 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" podStartSLOduration=1.737507995 podStartE2EDuration="1.737507995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:43.734057336 +0000 UTC m=+1573.696435776" watchObservedRunningTime="2026-11-03 20:05:43.737507995 +0000 UTC m=+1573.699886435" Nov 03 20:05:44.698169 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:44.698128 5111 generic.go:356] "Generic (PLEG): container finished" podID="11c72ddb-a97a-43b1-b26a-7814772a7d4c" containerID="51eaeb3fafc316ca0e9ed18103022e7e8016b21572ce144a39d3a27320ff5d15" exitCode=0 Nov 03 20:05:44.698716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:44.698211 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" event={"ID":"11c72ddb-a97a-43b1-b26a-7814772a7d4c","Type":"ContainerDied","Data":"51eaeb3fafc316ca0e9ed18103022e7e8016b21572ce144a39d3a27320ff5d15"} Nov 03 20:05:44.700300 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:44.700277 5111 generic.go:356] "Generic (PLEG): container finished" podID="7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f" containerID="af50fef6d72621b9b1cde5712f2677a5ebf06fca89a753025c4954e9032ecde9" exitCode=0 Nov 03 20:05:44.700409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:44.700361 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" event={"ID":"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f","Type":"ContainerDied","Data":"af50fef6d72621b9b1cde5712f2677a5ebf06fca89a753025c4954e9032ecde9"} Nov 03 20:05:44.886191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:44.886166 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" Nov 03 20:05:44.927512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:44.920850 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d76d5576-0a53-4653-bb66-c8e640e6f819-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76d5576-0a53-4653-bb66-c8e640e6f819-maas-api-openshift-ca\") pod \"d76d5576-0a53-4653-bb66-c8e640e6f819\" (UID: \"d76d5576-0a53-4653-bb66-c8e640e6f819\") " Nov 03 20:05:44.927512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:44.920983 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76d5576-0a53-4653-bb66-c8e640e6f819-kube-api-access-wnc5d\" (UniqueName: \"kubernetes.io/projected/d76d5576-0a53-4653-bb66-c8e640e6f819-kube-api-access-wnc5d\") pod \"d76d5576-0a53-4653-bb66-c8e640e6f819\" (UID: \"d76d5576-0a53-4653-bb66-c8e640e6f819\") " Nov 03 20:05:44.927512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:44.921060 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d76d5576-0a53-4653-bb66-c8e640e6f819-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76d5576-0a53-4653-bb66-c8e640e6f819-maas-api-xks-ca\") pod \"d76d5576-0a53-4653-bb66-c8e640e6f819\" (UID: \"d76d5576-0a53-4653-bb66-c8e640e6f819\") " Nov 03 20:05:44.947158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:44.944786 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d76d5576-0a53-4653-bb66-c8e640e6f819-maas-api-openshift-ca" pod "d76d5576-0a53-4653-bb66-c8e640e6f819" (UID: "d76d5576-0a53-4653-bb66-c8e640e6f819"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:44.954896 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:44.954854 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d76d5576-0a53-4653-bb66-c8e640e6f819-maas-api-xks-ca" pod "d76d5576-0a53-4653-bb66-c8e640e6f819" (UID: "d76d5576-0a53-4653-bb66-c8e640e6f819"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:44.966213 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:44.966155 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76d5576-0a53-4653-bb66-c8e640e6f819-kube-api-access-wnc5d" pod "d76d5576-0a53-4653-bb66-c8e640e6f819" (UID: "d76d5576-0a53-4653-bb66-c8e640e6f819"). InnerVolumeSpecName "kube-api-access-wnc5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:45.023646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:45.023605 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76d5576-0a53-4653-bb66-c8e640e6f819-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:45.023867 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:45.023847 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnc5d\" (UniqueName: \"kubernetes.io/projected/d76d5576-0a53-4653-bb66-c8e640e6f819-kube-api-access-wnc5d\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:45.024013 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:45.023872 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76d5576-0a53-4653-bb66-c8e640e6f819-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:45.710083 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:45.710043 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" event={"ID":"d76d5576-0a53-4653-bb66-c8e640e6f819","Type":"ContainerDied","Data":"a22c799645d20e8e40135fc978f5e3e4b153e1bcb3f881eade0246af567f8675"} Nov 03 20:05:45.710083 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:45.710068 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d" Nov 03 20:05:45.710580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:45.710104 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a22c799645d20e8e40135fc978f5e3e4b153e1bcb3f881eade0246af567f8675" Nov 03 20:05:45.974074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:45.974009 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" Nov 03 20:05:46.025340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.025309 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" Nov 03 20:05:46.049632 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.049598 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-maas-api-openshift-ca\") pod \"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f\" (UID: \"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f\") " Nov 03 20:05:46.049785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.049703 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-maas-api-xks-ca\") pod \"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f\" (UID: \"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f\") " Nov 03 20:05:46.049916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.049898 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-kube-api-access-rjxjz\" (UniqueName: \"kubernetes.io/projected/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-kube-api-access-rjxjz\") pod \"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f\" (UID: \"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f\") " Nov 03 20:05:46.049998 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.049977 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-maas-api-openshift-ca" pod "7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f" (UID: "7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:46.050944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.050526 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:46.054869 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.052603 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-kube-api-access-rjxjz" pod "7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f" (UID: "7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f"). InnerVolumeSpecName "kube-api-access-rjxjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:46.054869 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.053126 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-maas-api-xks-ca" pod "7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f" (UID: "7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:46.151480 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.151440 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c72ddb-a97a-43b1-b26a-7814772a7d4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c72ddb-a97a-43b1-b26a-7814772a7d4c-maas-api-openshift-ca\") pod \"11c72ddb-a97a-43b1-b26a-7814772a7d4c\" (UID: \"11c72ddb-a97a-43b1-b26a-7814772a7d4c\") " Nov 03 20:05:46.151686 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.151549 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11c72ddb-a97a-43b1-b26a-7814772a7d4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c72ddb-a97a-43b1-b26a-7814772a7d4c-maas-api-xks-ca\") pod \"11c72ddb-a97a-43b1-b26a-7814772a7d4c\" (UID: \"11c72ddb-a97a-43b1-b26a-7814772a7d4c\") " Nov 03 20:05:46.151771 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.151704 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c72ddb-a97a-43b1-b26a-7814772a7d4c-kube-api-access-7p6vt\" (UniqueName: \"kubernetes.io/projected/11c72ddb-a97a-43b1-b26a-7814772a7d4c-kube-api-access-7p6vt\") pod \"11c72ddb-a97a-43b1-b26a-7814772a7d4c\" (UID: \"11c72ddb-a97a-43b1-b26a-7814772a7d4c\") " Nov 03 20:05:46.151841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.151815 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c72ddb-a97a-43b1-b26a-7814772a7d4c-maas-api-openshift-ca" pod "11c72ddb-a97a-43b1-b26a-7814772a7d4c" (UID: "11c72ddb-a97a-43b1-b26a-7814772a7d4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:46.152652 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.152626 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c72ddb-a97a-43b1-b26a-7814772a7d4c-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:46.152751 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.152656 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:46.152751 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.152672 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjxjz\" (UniqueName: \"kubernetes.io/projected/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f-kube-api-access-rjxjz\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:46.153723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.153696 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c72ddb-a97a-43b1-b26a-7814772a7d4c-maas-api-xks-ca" pod "11c72ddb-a97a-43b1-b26a-7814772a7d4c" (UID: "11c72ddb-a97a-43b1-b26a-7814772a7d4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:46.153877 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.153851 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c72ddb-a97a-43b1-b26a-7814772a7d4c-kube-api-access-7p6vt" pod "11c72ddb-a97a-43b1-b26a-7814772a7d4c" (UID: "11c72ddb-a97a-43b1-b26a-7814772a7d4c"). InnerVolumeSpecName "kube-api-access-7p6vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:46.255091 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.255013 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p6vt\" (UniqueName: \"kubernetes.io/projected/11c72ddb-a97a-43b1-b26a-7814772a7d4c-kube-api-access-7p6vt\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:46.255091 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.255051 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c72ddb-a97a-43b1-b26a-7814772a7d4c-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:46.732676 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.732633 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" Nov 03 20:05:46.732676 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.732658 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv" event={"ID":"7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f","Type":"ContainerDied","Data":"112a8ea29f81fce65f43e31876deabb30ec5ba5f29659b5a3dd9971b6b366803"} Nov 03 20:05:46.733155 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.732744 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="112a8ea29f81fce65f43e31876deabb30ec5ba5f29659b5a3dd9971b6b366803" Nov 03 20:05:46.734703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.734680 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" Nov 03 20:05:46.734703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.734694 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll" event={"ID":"11c72ddb-a97a-43b1-b26a-7814772a7d4c","Type":"ContainerDied","Data":"5297f89f53fc8105dce06c3308aafa455482b650d711fb700f8379dab6c946a6"} Nov 03 20:05:46.734910 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:46.734726 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5297f89f53fc8105dce06c3308aafa455482b650d711fb700f8379dab6c946a6" Nov 03 20:05:47.756398 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.756357 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw"] Nov 03 20:05:47.757087 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.757068 5111 cpu_manager.go:414] "containerMap: removing container" podUID="d76d5576-0a53-4653-bb66-c8e640e6f819" containerName="revoke-keys" Nov 03 20:05:47.757087 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.757087 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76d5576-0a53-4653-bb66-c8e640e6f819" containerName="revoke-keys" Nov 03 20:05:47.757200 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.757107 5111 cpu_manager.go:414] "containerMap: removing container" podUID="7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f" containerName="revoke-keys" Nov 03 20:05:47.757200 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.757114 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f" containerName="revoke-keys" Nov 03 20:05:47.757200 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.757143 5111 cpu_manager.go:414] "containerMap: removing container" podUID="11c72ddb-a97a-43b1-b26a-7814772a7d4c" containerName="revoke-keys" Nov 03 20:05:47.757200 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.757148 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c72ddb-a97a-43b1-b26a-7814772a7d4c" containerName="revoke-keys" Nov 03 20:05:47.757389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.757264 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c72ddb-a97a-43b1-b26a-7814772a7d4c" containerName="revoke-keys" Nov 03 20:05:47.757389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.757278 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="d76d5576-0a53-4653-bb66-c8e640e6f819" containerName="revoke-keys" Nov 03 20:05:47.757389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.757286 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f" containerName="revoke-keys" Nov 03 20:05:47.761668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.761646 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw" Nov 03 20:05:47.767423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.767402 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:05:47.767590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.767579 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:05:47.767664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.767615 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:05:47.767737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.767665 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:05:47.768641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.768618 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw"] Nov 03 20:05:47.880345 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.880311 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1da6d9b3-554vw\" (UID: \"11c6efbc-ed2c-4664-b08b-1e8f97ac289a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw" Nov 03 20:05:47.880511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.880375 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1da6d9b3-554vw\" (UID: \"11c6efbc-ed2c-4664-b08b-1e8f97ac289a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw" Nov 03 20:05:47.880599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.880567 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sd2h\" (UniqueName: \"kubernetes.io/projected/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-kube-api-access-6sd2h\") pod \"maas-api-revoke-sub-1da6d9b3-554vw\" (UID: \"11c6efbc-ed2c-4664-b08b-1e8f97ac289a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw" Nov 03 20:05:47.981873 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.981835 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1da6d9b3-554vw\" (UID: \"11c6efbc-ed2c-4664-b08b-1e8f97ac289a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw" Nov 03 20:05:47.982052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.981900 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1da6d9b3-554vw\" (UID: \"11c6efbc-ed2c-4664-b08b-1e8f97ac289a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw" Nov 03 20:05:47.982052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.981998 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sd2h\" (UniqueName: \"kubernetes.io/projected/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-kube-api-access-6sd2h\") pod \"maas-api-revoke-sub-1da6d9b3-554vw\" (UID: \"11c6efbc-ed2c-4664-b08b-1e8f97ac289a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw" Nov 03 20:05:47.982624 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.982598 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1da6d9b3-554vw\" (UID: \"11c6efbc-ed2c-4664-b08b-1e8f97ac289a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw" Nov 03 20:05:47.984606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.984572 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1da6d9b3-554vw\" (UID: \"11c6efbc-ed2c-4664-b08b-1e8f97ac289a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw" Nov 03 20:05:47.990930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:47.990907 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sd2h\" (UniqueName: \"kubernetes.io/projected/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-kube-api-access-6sd2h\") pod \"maas-api-revoke-sub-1da6d9b3-554vw\" (UID: \"11c6efbc-ed2c-4664-b08b-1e8f97ac289a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw" Nov 03 20:05:48.098787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:48.098708 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw" Nov 03 20:05:48.392018 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:48.391979 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c6efbc_ed2c_4664_b08b_1e8f97ac289a.slice/crio-f3e35fe28cd34ee2659884a5da8315a2a244b3664fe17abf76f4c1bd244c88dd WatchSource:0}: Error finding container f3e35fe28cd34ee2659884a5da8315a2a244b3664fe17abf76f4c1bd244c88dd: Status 404 returned error can't find the container with id f3e35fe28cd34ee2659884a5da8315a2a244b3664fe17abf76f4c1bd244c88dd Nov 03 20:05:48.392229 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:48.392204 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw"] Nov 03 20:05:48.759612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:48.759577 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw" event={"ID":"11c6efbc-ed2c-4664-b08b-1e8f97ac289a","Type":"ContainerStarted","Data":"f3e35fe28cd34ee2659884a5da8315a2a244b3664fe17abf76f4c1bd244c88dd"} Nov 03 20:05:49.687771 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:49.687733 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4"] Nov 03 20:05:49.692377 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:49.692354 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" Nov 03 20:05:49.702907 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:49.702884 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4"] Nov 03 20:05:49.770570 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:49.770514 5111 generic.go:356] "Generic (PLEG): container finished" podID="11c6efbc-ed2c-4664-b08b-1e8f97ac289a" containerID="aca86c5d2619eafec6b4963a8ffb3a48f323e8d7ea224c7f1a278510f237be9f" exitCode=0 Nov 03 20:05:49.771191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:49.770601 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw" event={"ID":"11c6efbc-ed2c-4664-b08b-1e8f97ac289a","Type":"ContainerDied","Data":"aca86c5d2619eafec6b4963a8ffb3a48f323e8d7ea224c7f1a278510f237be9f"} Nov 03 20:05:49.814916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:49.814886 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5znt\" (UniqueName: \"kubernetes.io/projected/cfe7c70b-e981-4d26-a41e-934f655d2838-kube-api-access-j5znt\") pod \"maas-api-revoke-sub-d928b71f-ks6d4\" (UID: \"cfe7c70b-e981-4d26-a41e-934f655d2838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" Nov 03 20:05:49.815034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:49.814951 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe7c70b-e981-4d26-a41e-934f655d2838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d928b71f-ks6d4\" (UID: \"cfe7c70b-e981-4d26-a41e-934f655d2838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" Nov 03 20:05:49.815092 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:49.815031 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe7c70b-e981-4d26-a41e-934f655d2838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d928b71f-ks6d4\" (UID: \"cfe7c70b-e981-4d26-a41e-934f655d2838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" Nov 03 20:05:49.916767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:49.916730 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5znt\" (UniqueName: \"kubernetes.io/projected/cfe7c70b-e981-4d26-a41e-934f655d2838-kube-api-access-j5znt\") pod \"maas-api-revoke-sub-d928b71f-ks6d4\" (UID: \"cfe7c70b-e981-4d26-a41e-934f655d2838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" Nov 03 20:05:49.916951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:49.916796 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe7c70b-e981-4d26-a41e-934f655d2838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d928b71f-ks6d4\" (UID: \"cfe7c70b-e981-4d26-a41e-934f655d2838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" Nov 03 20:05:49.916951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:49.916828 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe7c70b-e981-4d26-a41e-934f655d2838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d928b71f-ks6d4\" (UID: \"cfe7c70b-e981-4d26-a41e-934f655d2838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" Nov 03 20:05:49.917641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:49.917617 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe7c70b-e981-4d26-a41e-934f655d2838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d928b71f-ks6d4\" (UID: \"cfe7c70b-e981-4d26-a41e-934f655d2838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" Nov 03 20:05:49.919282 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:49.919261 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe7c70b-e981-4d26-a41e-934f655d2838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d928b71f-ks6d4\" (UID: \"cfe7c70b-e981-4d26-a41e-934f655d2838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" Nov 03 20:05:49.926324 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:49.926300 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5znt\" (UniqueName: \"kubernetes.io/projected/cfe7c70b-e981-4d26-a41e-934f655d2838-kube-api-access-j5znt\") pod \"maas-api-revoke-sub-d928b71f-ks6d4\" (UID: \"cfe7c70b-e981-4d26-a41e-934f655d2838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" Nov 03 20:05:50.008925 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:50.008851 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" Nov 03 20:05:50.311853 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:50.311810 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4"] Nov 03 20:05:50.315705 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:50.315671 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe7c70b_e981_4d26_a41e_934f655d2838.slice/crio-ff5af8567ea8945a62012e8a7fabf94a9b2f2ae29bff47d8bc962e9685ce6831 WatchSource:0}: Error finding container ff5af8567ea8945a62012e8a7fabf94a9b2f2ae29bff47d8bc962e9685ce6831: Status 404 returned error can't find the container with id ff5af8567ea8945a62012e8a7fabf94a9b2f2ae29bff47d8bc962e9685ce6831 Nov 03 20:05:50.768470 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:50.768441 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp"] Nov 03 20:05:50.774217 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:50.774195 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" Nov 03 20:05:50.783964 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:50.783939 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" event={"ID":"cfe7c70b-e981-4d26-a41e-934f655d2838","Type":"ContainerStarted","Data":"cfd29e38c66babfecee912cf037add8e1f5f1596a6a6437b1b78763d70a061bd"} Nov 03 20:05:50.784074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:50.783989 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" event={"ID":"cfe7c70b-e981-4d26-a41e-934f655d2838","Type":"ContainerStarted","Data":"ff5af8567ea8945a62012e8a7fabf94a9b2f2ae29bff47d8bc962e9685ce6831"} Nov 03 20:05:50.790513 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:50.790405 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp"] Nov 03 20:05:50.828643 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:50.828526 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" podStartSLOduration=1.828507316 podStartE2EDuration="1.828507316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:50.819916688 +0000 UTC m=+1580.782295141" watchObservedRunningTime="2026-11-03 20:05:50.828507316 +0000 UTC m=+1580.790885760" Nov 03 20:05:50.934796 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:50.934763 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9lc5\" (UniqueName: \"kubernetes.io/projected/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-kube-api-access-k9lc5\") pod \"maas-api-revoke-sub-f360aa4f-l7vwp\" (UID: \"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" Nov 03 20:05:50.934954 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:50.934816 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f360aa4f-l7vwp\" (UID: \"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" Nov 03 20:05:50.935029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:50.934967 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f360aa4f-l7vwp\" (UID: \"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" Nov 03 20:05:50.965664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:50.965641 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw" Nov 03 20:05:51.035970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.035936 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9lc5\" (UniqueName: \"kubernetes.io/projected/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-kube-api-access-k9lc5\") pod \"maas-api-revoke-sub-f360aa4f-l7vwp\" (UID: \"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" Nov 03 20:05:51.036120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.035991 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f360aa4f-l7vwp\" (UID: \"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" Nov 03 20:05:51.036120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.036078 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f360aa4f-l7vwp\" (UID: \"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" Nov 03 20:05:51.036949 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.036915 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f360aa4f-l7vwp\" (UID: \"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" Nov 03 20:05:51.038625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.038595 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f360aa4f-l7vwp\" (UID: \"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" Nov 03 20:05:51.045623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.045593 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9lc5\" (UniqueName: \"kubernetes.io/projected/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-kube-api-access-k9lc5\") pod \"maas-api-revoke-sub-f360aa4f-l7vwp\" (UID: \"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" Nov 03 20:05:51.093127 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.093060 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" Nov 03 20:05:51.137656 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.137625 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-maas-api-openshift-ca\") pod \"11c6efbc-ed2c-4664-b08b-1e8f97ac289a\" (UID: \"11c6efbc-ed2c-4664-b08b-1e8f97ac289a\") " Nov 03 20:05:51.137884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.137864 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-maas-api-xks-ca\") pod \"11c6efbc-ed2c-4664-b08b-1e8f97ac289a\" (UID: \"11c6efbc-ed2c-4664-b08b-1e8f97ac289a\") " Nov 03 20:05:51.138003 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.137891 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-kube-api-access-6sd2h\" (UniqueName: \"kubernetes.io/projected/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-kube-api-access-6sd2h\") pod \"11c6efbc-ed2c-4664-b08b-1e8f97ac289a\" (UID: \"11c6efbc-ed2c-4664-b08b-1e8f97ac289a\") " Nov 03 20:05:51.138003 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.137978 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-maas-api-openshift-ca" pod "11c6efbc-ed2c-4664-b08b-1e8f97ac289a" (UID: "11c6efbc-ed2c-4664-b08b-1e8f97ac289a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:51.138618 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.138582 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:51.140047 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.140025 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-kube-api-access-6sd2h" pod "11c6efbc-ed2c-4664-b08b-1e8f97ac289a" (UID: "11c6efbc-ed2c-4664-b08b-1e8f97ac289a"). InnerVolumeSpecName "kube-api-access-6sd2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:51.140258 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.140234 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-maas-api-xks-ca" pod "11c6efbc-ed2c-4664-b08b-1e8f97ac289a" (UID: "11c6efbc-ed2c-4664-b08b-1e8f97ac289a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:51.240315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.240282 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:51.240315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.240315 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sd2h\" (UniqueName: \"kubernetes.io/projected/11c6efbc-ed2c-4664-b08b-1e8f97ac289a-kube-api-access-6sd2h\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:51.379611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.379579 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c"] Nov 03 20:05:51.380359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.380339 5111 cpu_manager.go:414] "containerMap: removing container" podUID="11c6efbc-ed2c-4664-b08b-1e8f97ac289a" containerName="revoke-keys" Nov 03 20:05:51.380459 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.380361 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c6efbc-ed2c-4664-b08b-1e8f97ac289a" containerName="revoke-keys" Nov 03 20:05:51.380601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.380586 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c6efbc-ed2c-4664-b08b-1e8f97ac289a" containerName="revoke-keys" Nov 03 20:05:51.384330 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.384311 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" Nov 03 20:05:51.394754 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.394723 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c"] Nov 03 20:05:51.401859 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:51.401837 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65482e2a_7eb1_4cd8_89fa_af7c7ad6c261.slice/crio-cd6600c13e859eb2fe3a72d7b40e6c05ad41e0c147a01f88cebee5b9fe8ec76c WatchSource:0}: Error finding container cd6600c13e859eb2fe3a72d7b40e6c05ad41e0c147a01f88cebee5b9fe8ec76c: Status 404 returned error can't find the container with id cd6600c13e859eb2fe3a72d7b40e6c05ad41e0c147a01f88cebee5b9fe8ec76c Nov 03 20:05:51.402775 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.402752 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp"] Nov 03 20:05:51.545286 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.545246 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvcjx\" (UniqueName: \"kubernetes.io/projected/245f0880-8769-447a-964a-0742bea73eb9-kube-api-access-mvcjx\") pod \"maas-api-revoke-sub-1fe55076-8cz8c\" (UID: \"245f0880-8769-447a-964a-0742bea73eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" Nov 03 20:05:51.545473 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.545381 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/245f0880-8769-447a-964a-0742bea73eb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe55076-8cz8c\" (UID: \"245f0880-8769-447a-964a-0742bea73eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" Nov 03 20:05:51.545573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.545472 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/245f0880-8769-447a-964a-0742bea73eb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe55076-8cz8c\" (UID: \"245f0880-8769-447a-964a-0742bea73eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" Nov 03 20:05:51.647047 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.647001 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/245f0880-8769-447a-964a-0742bea73eb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe55076-8cz8c\" (UID: \"245f0880-8769-447a-964a-0742bea73eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" Nov 03 20:05:51.647234 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.647204 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/245f0880-8769-447a-964a-0742bea73eb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe55076-8cz8c\" (UID: \"245f0880-8769-447a-964a-0742bea73eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" Nov 03 20:05:51.647415 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.647396 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvcjx\" (UniqueName: \"kubernetes.io/projected/245f0880-8769-447a-964a-0742bea73eb9-kube-api-access-mvcjx\") pod \"maas-api-revoke-sub-1fe55076-8cz8c\" (UID: \"245f0880-8769-447a-964a-0742bea73eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" Nov 03 20:05:51.648043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.647822 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/245f0880-8769-447a-964a-0742bea73eb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe55076-8cz8c\" (UID: \"245f0880-8769-447a-964a-0742bea73eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" Nov 03 20:05:51.650315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.650283 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/245f0880-8769-447a-964a-0742bea73eb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe55076-8cz8c\" (UID: \"245f0880-8769-447a-964a-0742bea73eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" Nov 03 20:05:51.655712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.655691 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvcjx\" (UniqueName: \"kubernetes.io/projected/245f0880-8769-447a-964a-0742bea73eb9-kube-api-access-mvcjx\") pod \"maas-api-revoke-sub-1fe55076-8cz8c\" (UID: \"245f0880-8769-447a-964a-0742bea73eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" Nov 03 20:05:51.727168 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.727130 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" Nov 03 20:05:51.794608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.794571 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw" event={"ID":"11c6efbc-ed2c-4664-b08b-1e8f97ac289a","Type":"ContainerDied","Data":"f3e35fe28cd34ee2659884a5da8315a2a244b3664fe17abf76f4c1bd244c88dd"} Nov 03 20:05:51.795052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.794627 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3e35fe28cd34ee2659884a5da8315a2a244b3664fe17abf76f4c1bd244c88dd" Nov 03 20:05:51.795052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.794688 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw" Nov 03 20:05:51.807294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.800989 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" event={"ID":"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261","Type":"ContainerStarted","Data":"7fb2f22d8c56444ecabf3bba9cfae798e784fb943c4d25ef4e1d01a26fbbbaf5"} Nov 03 20:05:51.807294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.801034 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" event={"ID":"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261","Type":"ContainerStarted","Data":"cd6600c13e859eb2fe3a72d7b40e6c05ad41e0c147a01f88cebee5b9fe8ec76c"} Nov 03 20:05:51.807294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.802931 5111 generic.go:356] "Generic (PLEG): container finished" podID="cfe7c70b-e981-4d26-a41e-934f655d2838" containerID="cfd29e38c66babfecee912cf037add8e1f5f1596a6a6437b1b78763d70a061bd" exitCode=0 Nov 03 20:05:51.807294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.802965 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" event={"ID":"cfe7c70b-e981-4d26-a41e-934f655d2838","Type":"ContainerDied","Data":"cfd29e38c66babfecee912cf037add8e1f5f1596a6a6437b1b78763d70a061bd"} Nov 03 20:05:51.828866 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:51.828820 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" podStartSLOduration=1.828807389 podStartE2EDuration="1.828807389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:51.82180414 +0000 UTC m=+1581.784182603" watchObservedRunningTime="2026-11-03 20:05:51.828807389 +0000 UTC m=+1581.791185827" Nov 03 20:05:52.019921 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.019889 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c"] Nov 03 20:05:52.020084 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:52.020057 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245f0880_8769_447a_964a_0742bea73eb9.slice/crio-187ce6013e7173fbc6773d421398c5b8eb8da3ac7a3ff9d00455976634ec4412 WatchSource:0}: Error finding container 187ce6013e7173fbc6773d421398c5b8eb8da3ac7a3ff9d00455976634ec4412: Status 404 returned error can't find the container with id 187ce6013e7173fbc6773d421398c5b8eb8da3ac7a3ff9d00455976634ec4412 Nov 03 20:05:52.216621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.216587 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s"] Nov 03 20:05:52.227825 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.227784 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.244334 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.244301 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s"] Nov 03 20:05:52.326269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.326189 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m"] Nov 03 20:05:52.330871 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.330847 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.343977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.343942 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m"] Nov 03 20:05:52.364680 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.364647 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.364825 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.364764 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.364961 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.364939 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.365111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.365011 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.365111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.365043 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.365111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.365084 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.365111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.365108 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzkbx\" (UniqueName: \"kubernetes.io/projected/b205487d-c7cb-4491-a6e5-c90771872d3c-kube-api-access-jzkbx\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.365575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.365387 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.365575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.365444 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b205487d-c7cb-4491-a6e5-c90771872d3c-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.466685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.466654 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.466908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.466884 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cff690-9d53-44d6-a1dc-3f96238ca235-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.467016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.466949 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.467016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.466997 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.467127 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.467047 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.467127 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.467098 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.467238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.467127 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzf9h\" (UniqueName: \"kubernetes.io/projected/87cff690-9d53-44d6-a1dc-3f96238ca235-kube-api-access-nzf9h\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.467296 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.467243 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.467348 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.467304 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.467643 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.467622 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.467715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.467687 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.467778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.467761 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.467847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.467825 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.467911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.467852 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.467911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.467858 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.467911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.467884 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzkbx\" (UniqueName: \"kubernetes.io/projected/b205487d-c7cb-4491-a6e5-c90771872d3c-kube-api-access-jzkbx\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.468059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.467933 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.468059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.467975 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.468059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.468011 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.468059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.468025 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.468263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.468214 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.468317 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.468273 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b205487d-c7cb-4491-a6e5-c90771872d3c-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.469055 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.469034 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b205487d-c7cb-4491-a6e5-c90771872d3c-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.470326 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.470304 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.471693 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.471651 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.480291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.480272 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.483572 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.483509 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzkbx\" (UniqueName: \"kubernetes.io/projected/b205487d-c7cb-4491-a6e5-c90771872d3c-kube-api-access-jzkbx\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.502525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.502492 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq"] Nov 03 20:05:52.530703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.530664 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.531913 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.531874 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq"] Nov 03 20:05:52.544650 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.544624 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb"] Nov 03 20:05:52.552792 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.552758 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb" Nov 03 20:05:52.561159 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.561117 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:05:52.568285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.568246 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb"] Nov 03 20:05:52.570040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.569314 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.570040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.569392 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.570040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.569438 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.570040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.569485 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.570040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.569592 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.570040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.569626 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cff690-9d53-44d6-a1dc-3f96238ca235-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.570040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.569671 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.570040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.569731 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.570040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.569775 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzf9h\" (UniqueName: \"kubernetes.io/projected/87cff690-9d53-44d6-a1dc-3f96238ca235-kube-api-access-nzf9h\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.574313 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.571515 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.574313 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.571825 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.574313 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.572106 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.574313 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.572334 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.574313 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.573831 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cff690-9d53-44d6-a1dc-3f96238ca235-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.580801 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.579495 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.582506 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.582463 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.591368 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.591307 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.592793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.592773 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzf9h\" (UniqueName: \"kubernetes.io/projected/87cff690-9d53-44d6-a1dc-3f96238ca235-kube-api-access-nzf9h\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.654876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.654561 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:05:52.672641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.671961 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.672641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.671994 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7667p\" (UniqueName: \"kubernetes.io/projected/75b8d7d9-7e56-4f0d-85f8-d5449b803007-kube-api-access-7667p\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.672641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.672023 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d40208cf-757c-4b13-a2f6-722e534b1f63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a121a291-jpjcb\" (UID: \"d40208cf-757c-4b13-a2f6-722e534b1f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb" Nov 03 20:05:52.672641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.672087 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.672641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.672104 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.672641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.672122 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.672641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.672224 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.672641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.672254 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj2mx\" (UniqueName: \"kubernetes.io/projected/d40208cf-757c-4b13-a2f6-722e534b1f63-kube-api-access-xj2mx\") pod \"maas-api-revoke-sub-a121a291-jpjcb\" (UID: \"d40208cf-757c-4b13-a2f6-722e534b1f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb" Nov 03 20:05:52.672641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.672319 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.672641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.672338 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d40208cf-757c-4b13-a2f6-722e534b1f63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a121a291-jpjcb\" (UID: \"d40208cf-757c-4b13-a2f6-722e534b1f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb" Nov 03 20:05:52.672641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.672385 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.672641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.672438 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.776165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.774081 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.776165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.774145 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d40208cf-757c-4b13-a2f6-722e534b1f63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a121a291-jpjcb\" (UID: \"d40208cf-757c-4b13-a2f6-722e534b1f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb" Nov 03 20:05:52.776165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.774219 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.776165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.774326 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.776165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.774426 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.776165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.774462 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7667p\" (UniqueName: \"kubernetes.io/projected/75b8d7d9-7e56-4f0d-85f8-d5449b803007-kube-api-access-7667p\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.776165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.774506 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d40208cf-757c-4b13-a2f6-722e534b1f63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a121a291-jpjcb\" (UID: \"d40208cf-757c-4b13-a2f6-722e534b1f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb" Nov 03 20:05:52.776165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.774628 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.776165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.774656 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.776165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.774689 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.776165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.774833 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.776165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.774885 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj2mx\" (UniqueName: \"kubernetes.io/projected/d40208cf-757c-4b13-a2f6-722e534b1f63-kube-api-access-xj2mx\") pod \"maas-api-revoke-sub-a121a291-jpjcb\" (UID: \"d40208cf-757c-4b13-a2f6-722e534b1f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb" Nov 03 20:05:52.776165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.775792 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.788581 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.786816 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d40208cf-757c-4b13-a2f6-722e534b1f63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a121a291-jpjcb\" (UID: \"d40208cf-757c-4b13-a2f6-722e534b1f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb" Nov 03 20:05:52.788581 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.787412 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.788581 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.787454 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.788581 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.787719 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.788581 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.788268 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.788581 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.788527 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:52.811197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.802037 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d40208cf-757c-4b13-a2f6-722e534b1f63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a121a291-jpjcb\" (UID: \"d40208cf-757c-4b13-a2f6-722e534b1f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb" Nov 03 20:05:52.906559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.898288 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj2mx\" (UniqueName: \"kubernetes.io/projected/d40208cf-757c-4b13-a2f6-722e534b1f63-kube-api-access-xj2mx\") pod \"maas-api-revoke-sub-a121a291-jpjcb\" (UID: \"d40208cf-757c-4b13-a2f6-722e534b1f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb" Nov 03 20:05:52.906559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:52.902199 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:53.135595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:53.089949 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s"] Nov 03 20:05:53.181554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:53.180210 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb" Nov 03 20:05:53.268739 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:53.234780 5111 generic.go:356] "Generic (PLEG): container finished" podID="65482e2a-7eb1-4cd8-89fa-af7c7ad6c261" containerID="7fb2f22d8c56444ecabf3bba9cfae798e784fb943c4d25ef4e1d01a26fbbbaf5" exitCode=0 Nov 03 20:05:53.268739 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:53.234894 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" event={"ID":"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261","Type":"ContainerDied","Data":"7fb2f22d8c56444ecabf3bba9cfae798e784fb943c4d25ef4e1d01a26fbbbaf5"} Nov 03 20:05:53.533274 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:53.454140 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" event={"ID":"245f0880-8769-447a-964a-0742bea73eb9","Type":"ContainerStarted","Data":"e11653af039a9c83b227ef10618355af74e6af16f487fff068db6dbcfaab22ab"} Nov 03 20:05:53.533274 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:53.454222 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" event={"ID":"245f0880-8769-447a-964a-0742bea73eb9","Type":"ContainerStarted","Data":"187ce6013e7173fbc6773d421398c5b8eb8da3ac7a3ff9d00455976634ec4412"} Nov 03 20:05:53.736130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:53.712884 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:53.736130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:53.712984 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:53.736130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:53.713027 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:53.894465 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:53.894390 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:54.005845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:53.985161 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7667p\" (UniqueName: \"kubernetes.io/projected/75b8d7d9-7e56-4f0d-85f8-d5449b803007-kube-api-access-7667p\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:54.087447 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.087414 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:05:54.266384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.142801 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:54.266384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.219816 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 20:05:54.266384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.219876 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:05:54.266384 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:54.223012 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb205487d_c7cb_4491_a6e5_c90771872d3c.slice/crio-9f22b791d73ab540bd7cd9575b39fe3585efc1d3e92679674acf7eb184127e91 WatchSource:0}: Error finding container 9f22b791d73ab540bd7cd9575b39fe3585efc1d3e92679674acf7eb184127e91: Status 404 returned error can't find the container with id 9f22b791d73ab540bd7cd9575b39fe3585efc1d3e92679674acf7eb184127e91 Nov 03 20:05:54.266384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.226947 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:54.266384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.265567 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:54.266384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.265629 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:05:54.266384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.235761 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 20:05:54.268342 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.267206 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:54.350557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.350346 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:54.418220 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.418181 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m"] Nov 03 20:05:54.432375 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.432332 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:54.432932 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.432901 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 20:05:54.475136 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.474682 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:54.512635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.485860 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" podStartSLOduration=3.485842081 podStartE2EDuration="3.485842081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:53.703605826 +0000 UTC m=+1583.665984282" watchObservedRunningTime="2026-11-03 20:05:54.485842081 +0000 UTC m=+1584.448220521" Nov 03 20:05:54.538555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.513340 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:05:54.673646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.673103 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" event={"ID":"87cff690-9d53-44d6-a1dc-3f96238ca235","Type":"ContainerStarted","Data":"53f5703358a03e2fcc0d82de5fe63f2dee7ea86289a8e9f2b90e52419aa07aa9"} Nov 03 20:05:54.673646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.673227 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:54.701176 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.700709 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:54.711246 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.711210 5111 generic.go:356] "Generic (PLEG): container finished" podID="245f0880-8769-447a-964a-0742bea73eb9" containerID="e11653af039a9c83b227ef10618355af74e6af16f487fff068db6dbcfaab22ab" exitCode=0 Nov 03 20:05:54.711642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.711614 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" event={"ID":"245f0880-8769-447a-964a-0742bea73eb9","Type":"ContainerDied","Data":"e11653af039a9c83b227ef10618355af74e6af16f487fff068db6dbcfaab22ab"} Nov 03 20:05:54.750558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:54.734028 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" event={"ID":"b205487d-c7cb-4491-a6e5-c90771872d3c","Type":"ContainerStarted","Data":"9f22b791d73ab540bd7cd9575b39fe3585efc1d3e92679674acf7eb184127e91"} Nov 03 20:05:55.272621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.266069 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:05:55.283009 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.282962 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:55.328191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.328154 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:05:55.328478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.328456 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:05:55.393628 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.350879 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:05:55.393628 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.374552 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:05:55.393628 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.379034 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:55.458650 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.394046 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 20:05:55.458650 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.394698 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:55.513561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.488422 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded" Nov 03 20:05:55.778563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.505164 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:55.778563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.517930 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:05:55.778563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.766822 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:05:55.778563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.767203 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:05:55.778563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.767407 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:05:55.961568 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.952292 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:05:55.977617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.791115 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:56.061557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.791216 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 20:05:56.232034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:55.792190 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:56.236556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:56.205684 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:05:56.315732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:56.265727 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:05:56.315732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:56.265854 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:05:56.605012 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:56.604973 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:05:56.643684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:56.643630 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 20:05:56.723520 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:56.723474 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:05:56.790784 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:56.790719 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 20:05:56.857776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:56.850668 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 20:05:56.857776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:56.852298 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:05:56.857776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:56.852357 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" podUID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:57.000710 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:57.000657 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:57.152717 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:57.145057 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:05:57.152717 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:57.145373 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:05:57.335718 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:57.282818 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:05:57.421980 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:57.421932 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 20:05:57.422402 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:57.422372 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:05:57.426625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:57.345850 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:57.438556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:57.430805 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Nov 03 20:05:57.713616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:57.710094 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:05:57.717760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:57.717716 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 20:05:58.127712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:58.113450 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 20:05:58.127712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:58.114861 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:05:58.191251 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:58.188684 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:58.191251 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:58.189827 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 20:05:58.379760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:58.379718 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:05:58.407585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:58.380598 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:05:58.485543 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:58.435655 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" podUID="55381c03-06d5-4cdb-841f-265b16b1d8f8" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.70:50052\" within 1s: context deadline exceeded" Nov 03 20:05:58.485800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:58.435712 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:58.504443 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:58.436961 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:05:58.504878 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:05:58.453005 5111 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.187s" Nov 03 20:05:58.521682 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:58.514690 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" Nov 03 20:05:58.521682 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:58.472231 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-rvrqd" podUID="e49d0823-94c9-43a3-9963-72ed46e78ae8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 03 20:05:58.521682 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:58.472898 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 03 20:05:58.521682 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:05:58.480778 5111 logging.go:55] [core] [Channel #716 SubChannel #717]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.70:50052", ServerName: "10.132.0.70:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.70:50052: operation was canceled" Nov 03 20:05:58.522393 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:58.453050 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:58.762784 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:58.755718 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" podUID="1e292de6-b36f-481b-9d69-1961e7d8026b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:58.937869 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:58.823894 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:59.146616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:59.121229 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfe7c70b-e981-4d26-a41e-934f655d2838-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe7c70b-e981-4d26-a41e-934f655d2838-maas-api-openshift-ca\") pod \"cfe7c70b-e981-4d26-a41e-934f655d2838\" (UID: \"cfe7c70b-e981-4d26-a41e-934f655d2838\") " Nov 03 20:05:59.146616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:59.121369 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe7c70b-e981-4d26-a41e-934f655d2838-kube-api-access-j5znt\" (UniqueName: \"kubernetes.io/projected/cfe7c70b-e981-4d26-a41e-934f655d2838-kube-api-access-j5znt\") pod \"cfe7c70b-e981-4d26-a41e-934f655d2838\" (UID: \"cfe7c70b-e981-4d26-a41e-934f655d2838\") " Nov 03 20:05:59.146616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:59.121634 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfe7c70b-e981-4d26-a41e-934f655d2838-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe7c70b-e981-4d26-a41e-934f655d2838-maas-api-xks-ca\") pod \"cfe7c70b-e981-4d26-a41e-934f655d2838\" (UID: \"cfe7c70b-e981-4d26-a41e-934f655d2838\") " Nov 03 20:05:59.146616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:59.141734 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:59.146616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:59.143152 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 20:05:59.146616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:59.145121 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe7c70b-e981-4d26-a41e-934f655d2838-maas-api-openshift-ca" pod "cfe7c70b-e981-4d26-a41e-934f655d2838" (UID: "cfe7c70b-e981-4d26-a41e-934f655d2838"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:59.353749 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:59.330484 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe7c70b-e981-4d26-a41e-934f655d2838-kube-api-access-j5znt" pod "cfe7c70b-e981-4d26-a41e-934f655d2838" (UID: "cfe7c70b-e981-4d26-a41e-934f655d2838"). InnerVolumeSpecName "kube-api-access-j5znt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:59.401327 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:59.394129 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:59.407695 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:59.407651 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:59.480579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:59.480509 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c8tdm" podUID="989fbc44-448a-4608-904f-b265dcaccce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:59.524555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:59.521375 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:05:59.580558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:59.523037 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:05:59.605722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:05:59.605625 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:06:00.183831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.183787 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:00.207632 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.207585 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded" Nov 03 20:06:00.341417 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.341373 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:06:00.342222 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.342189 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:00.412038 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.412000 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:06:00.433230 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.433173 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 20:06:00.517706 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.517662 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:00.541567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.519943 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe7c70b-e981-4d26-a41e-934f655d2838-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:00.541567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.519970 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5znt\" (UniqueName: \"kubernetes.io/projected/cfe7c70b-e981-4d26-a41e-934f655d2838-kube-api-access-j5znt\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:00.542644 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.542202 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:06:00.585013 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:06:00.581323 5111 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.076s" Nov 03 20:06:00.661262 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.661211 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:06:00.661626 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.661571 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:06:00.661798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.661774 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:06:00.703791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.700450 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfe7c70b-e981-4d26-a41e-934f655d2838-maas-api-xks-ca" pod "cfe7c70b-e981-4d26-a41e-934f655d2838" (UID: "cfe7c70b-e981-4d26-a41e-934f655d2838"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:00.760565 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.751715 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:00.811848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.807284 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:06:00.811848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.807461 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:06:00.811848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.807497 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:06:00.811848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.807553 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:06:00.811848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.807588 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:06:00.811848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.807639 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:06:00.811848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.807670 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:06:00.811848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.807699 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:06:00.811848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.807736 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:06:00.811848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.807768 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:06:00.811848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.807799 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:06:00.811848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.807829 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:06:00.921468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.921419 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe7c70b-e981-4d26-a41e-934f655d2838-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:00.933700 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.933172 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:06:00.943553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.935996 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" event={"ID":"87cff690-9d53-44d6-a1dc-3f96238ca235","Type":"ContainerStarted","Data":"e62ecfa20290885ff5a59d4f58917bea7d380b6f21ce5da9bdf8187bb7dbf786"} Nov 03 20:06:00.943553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:00.940982 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" Nov 03 20:06:01.026653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.026595 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-maas-api-xks-ca\") pod \"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261\" (UID: \"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261\") " Nov 03 20:06:01.026940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.026916 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-maas-api-openshift-ca\") pod \"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261\" (UID: \"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261\") " Nov 03 20:06:01.027075 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.027057 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-kube-api-access-k9lc5\" (UniqueName: \"kubernetes.io/projected/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-kube-api-access-k9lc5\") pod \"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261\" (UID: \"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261\") " Nov 03 20:06:01.031300 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.031269 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-maas-api-openshift-ca" pod "65482e2a-7eb1-4cd8-89fa-af7c7ad6c261" (UID: "65482e2a-7eb1-4cd8-89fa-af7c7ad6c261"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:01.033693 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.033020 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:01.037265 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.036988 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-kube-api-access-k9lc5" pod "65482e2a-7eb1-4cd8-89fa-af7c7ad6c261" (UID: "65482e2a-7eb1-4cd8-89fa-af7c7ad6c261"). InnerVolumeSpecName "kube-api-access-k9lc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:01.039966 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.039283 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-maas-api-xks-ca" pod "65482e2a-7eb1-4cd8-89fa-af7c7ad6c261" (UID: "65482e2a-7eb1-4cd8-89fa-af7c7ad6c261"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:01.121919 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:06:01.121875 5111 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe7c70b_e981_4d26_a41e_934f655d2838.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:06:01.133080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.130156 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb"] Nov 03 20:06:01.139985 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.139953 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9lc5\" (UniqueName: \"kubernetes.io/projected/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-kube-api-access-k9lc5\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:01.140172 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.139991 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:01.221270 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.221159 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq"] Nov 03 20:06:01.235340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.234055 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz"] Nov 03 20:06:01.236472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.236374 5111 cpu_manager.go:414] "containerMap: removing container" podUID="65482e2a-7eb1-4cd8-89fa-af7c7ad6c261" containerName="revoke-keys" Nov 03 20:06:01.236472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.236406 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65482e2a-7eb1-4cd8-89fa-af7c7ad6c261" containerName="revoke-keys" Nov 03 20:06:01.236472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.236436 5111 cpu_manager.go:414] "containerMap: removing container" podUID="cfe7c70b-e981-4d26-a41e-934f655d2838" containerName="revoke-keys" Nov 03 20:06:01.236472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.236444 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe7c70b-e981-4d26-a41e-934f655d2838" containerName="revoke-keys" Nov 03 20:06:01.236800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.236710 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe7c70b-e981-4d26-a41e-934f655d2838" containerName="revoke-keys" Nov 03 20:06:01.236800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.236747 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="65482e2a-7eb1-4cd8-89fa-af7c7ad6c261" containerName="revoke-keys" Nov 03 20:06:01.305211 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.243386 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz"] Nov 03 20:06:01.305211 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.243494 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz" Nov 03 20:06:01.424705 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.415662 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zspc\" (UniqueName: \"kubernetes.io/projected/60956010-1d2b-4873-bff8-384c42f66e4d-kube-api-access-8zspc\") pod \"maas-api-revoke-sub-20b93cf5-r4kvz\" (UID: \"60956010-1d2b-4873-bff8-384c42f66e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz" Nov 03 20:06:01.426690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.425513 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60956010-1d2b-4873-bff8-384c42f66e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20b93cf5-r4kvz\" (UID: \"60956010-1d2b-4873-bff8-384c42f66e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz" Nov 03 20:06:01.427014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.426980 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60956010-1d2b-4873-bff8-384c42f66e4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20b93cf5-r4kvz\" (UID: \"60956010-1d2b-4873-bff8-384c42f66e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz" Nov 03 20:06:01.428756 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.428702 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" podStartSLOduration=9.428684827 podStartE2EDuration="9.428684827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:01.154684864 +0000 UTC m=+1591.117063297" watchObservedRunningTime="2026-11-03 20:06:01.428684827 +0000 UTC m=+1591.391063268" Nov 03 20:06:01.529413 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.529321 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zspc\" (UniqueName: \"kubernetes.io/projected/60956010-1d2b-4873-bff8-384c42f66e4d-kube-api-access-8zspc\") pod \"maas-api-revoke-sub-20b93cf5-r4kvz\" (UID: \"60956010-1d2b-4873-bff8-384c42f66e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz" Nov 03 20:06:01.530965 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.530936 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60956010-1d2b-4873-bff8-384c42f66e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20b93cf5-r4kvz\" (UID: \"60956010-1d2b-4873-bff8-384c42f66e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz" Nov 03 20:06:01.531197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.531176 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60956010-1d2b-4873-bff8-384c42f66e4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20b93cf5-r4kvz\" (UID: \"60956010-1d2b-4873-bff8-384c42f66e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz" Nov 03 20:06:01.534338 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.534316 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60956010-1d2b-4873-bff8-384c42f66e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20b93cf5-r4kvz\" (UID: \"60956010-1d2b-4873-bff8-384c42f66e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz" Nov 03 20:06:01.538857 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.538833 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60956010-1d2b-4873-bff8-384c42f66e4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20b93cf5-r4kvz\" (UID: \"60956010-1d2b-4873-bff8-384c42f66e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz" Nov 03 20:06:01.678563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.676996 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:06:01.713869 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.685521 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:06:01.713869 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.685595 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" podUID="87cff690-9d53-44d6-a1dc-3f96238ca235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 20:06:01.713869 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.695306 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zspc\" (UniqueName: \"kubernetes.io/projected/60956010-1d2b-4873-bff8-384c42f66e4d-kube-api-access-8zspc\") pod \"maas-api-revoke-sub-20b93cf5-r4kvz\" (UID: \"60956010-1d2b-4873-bff8-384c42f66e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz" Nov 03 20:06:01.726980 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.726928 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz" Nov 03 20:06:01.825945 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.825908 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" Nov 03 20:06:01.913823 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.894096 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/245f0880-8769-447a-964a-0742bea73eb9-kube-api-access-mvcjx\" (UniqueName: \"kubernetes.io/projected/245f0880-8769-447a-964a-0742bea73eb9-kube-api-access-mvcjx\") pod \"245f0880-8769-447a-964a-0742bea73eb9\" (UID: \"245f0880-8769-447a-964a-0742bea73eb9\") " Nov 03 20:06:01.913823 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.894356 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/245f0880-8769-447a-964a-0742bea73eb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/245f0880-8769-447a-964a-0742bea73eb9-maas-api-xks-ca\") pod \"245f0880-8769-447a-964a-0742bea73eb9\" (UID: \"245f0880-8769-447a-964a-0742bea73eb9\") " Nov 03 20:06:01.913823 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.894450 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/245f0880-8769-447a-964a-0742bea73eb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/245f0880-8769-447a-964a-0742bea73eb9-maas-api-openshift-ca\") pod \"245f0880-8769-447a-964a-0742bea73eb9\" (UID: \"245f0880-8769-447a-964a-0742bea73eb9\") " Nov 03 20:06:01.919869 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:01.919820 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245f0880-8769-447a-964a-0742bea73eb9-maas-api-openshift-ca" pod "245f0880-8769-447a-964a-0742bea73eb9" (UID: "245f0880-8769-447a-964a-0742bea73eb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:02.034497 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.034456 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/245f0880-8769-447a-964a-0742bea73eb9-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:02.041843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.041756 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" event={"ID":"65482e2a-7eb1-4cd8-89fa-af7c7ad6c261","Type":"ContainerDied","Data":"cd6600c13e859eb2fe3a72d7b40e6c05ad41e0c147a01f88cebee5b9fe8ec76c"} Nov 03 20:06:02.041843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.041824 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd6600c13e859eb2fe3a72d7b40e6c05ad41e0c147a01f88cebee5b9fe8ec76c" Nov 03 20:06:02.042047 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.041906 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp" Nov 03 20:06:02.116889 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.116855 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" event={"ID":"245f0880-8769-447a-964a-0742bea73eb9","Type":"ContainerDied","Data":"187ce6013e7173fbc6773d421398c5b8eb8da3ac7a3ff9d00455976634ec4412"} Nov 03 20:06:02.117130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.117113 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="187ce6013e7173fbc6773d421398c5b8eb8da3ac7a3ff9d00455976634ec4412" Nov 03 20:06:02.117278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.117266 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c" Nov 03 20:06:02.120278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.120250 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" event={"ID":"b205487d-c7cb-4491-a6e5-c90771872d3c","Type":"ContainerStarted","Data":"7337209cb74a6ef11f414a99069c2b9031a1b4440c15df98c39088b93ce9983d"} Nov 03 20:06:02.128175 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.128129 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/245f0880-8769-447a-964a-0742bea73eb9-maas-api-xks-ca" pod "245f0880-8769-447a-964a-0742bea73eb9" (UID: "245f0880-8769-447a-964a-0742bea73eb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:02.129690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.129655 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" event={"ID":"75b8d7d9-7e56-4f0d-85f8-d5449b803007","Type":"ContainerStarted","Data":"0b982f7e87145ffd194b096b125bb399f8af4ffa676ab18a9f4ad56372b852c2"} Nov 03 20:06:02.129824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.129720 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" event={"ID":"75b8d7d9-7e56-4f0d-85f8-d5449b803007","Type":"ContainerStarted","Data":"53df79d223a01581d677d98c3736e432368cd69227f15db0cadd01edb4704ff5"} Nov 03 20:06:02.132060 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.132016 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245f0880-8769-447a-964a-0742bea73eb9-kube-api-access-mvcjx" pod "245f0880-8769-447a-964a-0742bea73eb9" (UID: "245f0880-8769-447a-964a-0742bea73eb9"). InnerVolumeSpecName "kube-api-access-mvcjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:02.136285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.136220 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvcjx\" (UniqueName: \"kubernetes.io/projected/245f0880-8769-447a-964a-0742bea73eb9-kube-api-access-mvcjx\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:02.136285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.136249 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/245f0880-8769-447a-964a-0742bea73eb9-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:02.137465 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.137442 5111 generic.go:356] "Generic (PLEG): container finished" podID="d40208cf-757c-4b13-a2f6-722e534b1f63" containerID="0cb132693d178a51d1a9e22b832844247c2690e85bd0eaa00c6c58d1eeea26fc" exitCode=0 Nov 03 20:06:02.137803 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.137691 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb" event={"ID":"d40208cf-757c-4b13-a2f6-722e534b1f63","Type":"ContainerDied","Data":"0cb132693d178a51d1a9e22b832844247c2690e85bd0eaa00c6c58d1eeea26fc"} Nov 03 20:06:02.137956 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.137942 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb" event={"ID":"d40208cf-757c-4b13-a2f6-722e534b1f63","Type":"ContainerStarted","Data":"1149b64e1d5456310a5ffcb28e5574db88ea4e309774798db9d1089cb8f6183d"} Nov 03 20:06:02.141228 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.141191 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" Nov 03 20:06:02.141939 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.141905 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4" event={"ID":"cfe7c70b-e981-4d26-a41e-934f655d2838","Type":"ContainerDied","Data":"ff5af8567ea8945a62012e8a7fabf94a9b2f2ae29bff47d8bc962e9685ce6831"} Nov 03 20:06:02.142086 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.142072 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff5af8567ea8945a62012e8a7fabf94a9b2f2ae29bff47d8bc962e9685ce6831" Nov 03 20:06:02.340767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.340705 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" podStartSLOduration=10.340684123 podStartE2EDuration="10.340684123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:02.166975083 +0000 UTC m=+1592.129353523" watchObservedRunningTime="2026-11-03 20:06:02.340684123 +0000 UTC m=+1592.303062563" Nov 03 20:06:02.451739 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.451670 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" podStartSLOduration=10.451650328 podStartE2EDuration="10.451650328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:02.275176862 +0000 UTC m=+1592.237555314" watchObservedRunningTime="2026-11-03 20:06:02.451650328 +0000 UTC m=+1592.414028769" Nov 03 20:06:02.591623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.562934 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:06:02.608556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.595035 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 20:06:02.608556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.595095 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" podUID="b205487d-c7cb-4491-a6e5-c90771872d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 20:06:02.688556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.639621 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq"] Nov 03 20:06:02.688556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.640818 5111 cpu_manager.go:414] "containerMap: removing container" podUID="245f0880-8769-447a-964a-0742bea73eb9" containerName="revoke-keys" Nov 03 20:06:02.688556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.640843 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245f0880-8769-447a-964a-0742bea73eb9" containerName="revoke-keys" Nov 03 20:06:02.688556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.641022 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="245f0880-8769-447a-964a-0742bea73eb9" containerName="revoke-keys" Nov 03 20:06:02.688556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.663018 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:06:02.688556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.663092 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" podUID="87cff690-9d53-44d6-a1dc-3f96238ca235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 20:06:02.688556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.682885 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" Nov 03 20:06:02.909557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.884922 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2050bea5-d734-487f-ba88-5eb6b4d67f81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51249b31-77vgq\" (UID: \"2050bea5-d734-487f-ba88-5eb6b4d67f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" Nov 03 20:06:02.913356 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.913310 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2050bea5-d734-487f-ba88-5eb6b4d67f81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51249b31-77vgq\" (UID: \"2050bea5-d734-487f-ba88-5eb6b4d67f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" Nov 03 20:06:02.919565 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.894622 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq"] Nov 03 20:06:02.931169 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:02.930684 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lq9v\" (UniqueName: \"kubernetes.io/projected/2050bea5-d734-487f-ba88-5eb6b4d67f81-kube-api-access-4lq9v\") pod \"maas-api-revoke-sub-51249b31-77vgq\" (UID: \"2050bea5-d734-487f-ba88-5eb6b4d67f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" Nov 03 20:06:03.108911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:03.108879 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:06:03.264637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:03.221101 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lq9v\" (UniqueName: \"kubernetes.io/projected/2050bea5-d734-487f-ba88-5eb6b4d67f81-kube-api-access-4lq9v\") pod \"maas-api-revoke-sub-51249b31-77vgq\" (UID: \"2050bea5-d734-487f-ba88-5eb6b4d67f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" Nov 03 20:06:03.264637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:03.221295 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2050bea5-d734-487f-ba88-5eb6b4d67f81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51249b31-77vgq\" (UID: \"2050bea5-d734-487f-ba88-5eb6b4d67f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" Nov 03 20:06:03.264637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:03.221479 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2050bea5-d734-487f-ba88-5eb6b4d67f81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51249b31-77vgq\" (UID: \"2050bea5-d734-487f-ba88-5eb6b4d67f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" Nov 03 20:06:03.264637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:03.249163 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:06:03.264637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:03.249225 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" podUID="75b8d7d9-7e56-4f0d-85f8-d5449b803007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 20:06:03.354390 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:03.354351 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2050bea5-d734-487f-ba88-5eb6b4d67f81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51249b31-77vgq\" (UID: \"2050bea5-d734-487f-ba88-5eb6b4d67f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" Nov 03 20:06:03.420555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:03.393716 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz"] Nov 03 20:06:03.495566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:03.494046 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2050bea5-d734-487f-ba88-5eb6b4d67f81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51249b31-77vgq\" (UID: \"2050bea5-d734-487f-ba88-5eb6b4d67f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" Nov 03 20:06:03.605605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:03.594947 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 20:06:03.605605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:03.595018 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" podUID="b205487d-c7cb-4491-a6e5-c90771872d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 20:06:03.609860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:03.609828 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz" event={"ID":"60956010-1d2b-4873-bff8-384c42f66e4d","Type":"ContainerStarted","Data":"7187ef42bb9a915950a830f94f3d80105f3544222c11bb874972b9cc907ff3b4"} Nov 03 20:06:03.693557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:03.691451 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lq9v\" (UniqueName: \"kubernetes.io/projected/2050bea5-d734-487f-ba88-5eb6b4d67f81-kube-api-access-4lq9v\") pod \"maas-api-revoke-sub-51249b31-77vgq\" (UID: \"2050bea5-d734-487f-ba88-5eb6b4d67f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" Nov 03 20:06:03.810908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:03.802868 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:06:03.810908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:03.809858 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" podUID="87cff690-9d53-44d6-a1dc-3f96238ca235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 20:06:03.992958 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:03.991815 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" Nov 03 20:06:04.130461 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:04.124731 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:06:04.130461 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:04.124788 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" podUID="75b8d7d9-7e56-4f0d-85f8-d5449b803007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 20:06:04.240551 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:04.235327 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:04.471555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:04.470839 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 20:06:04.636847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:04.636802 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Nov 03 20:06:04.666313 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:04.637765 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:04.858204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:04.722629 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 20:06:04.858204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:04.732105 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 20:06:04.858204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:04.740045 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" podUID="b205487d-c7cb-4491-a6e5-c90771872d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 20:06:04.858204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:04.732330 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:06:04.858204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:04.740095 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" podUID="87cff690-9d53-44d6-a1dc-3f96238ca235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 20:06:04.858204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:04.806554 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:04.858204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:04.818139 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:04.859271 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:04.858977 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:05.135617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:05.130249 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:06:05.135617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:05.130308 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" podUID="75b8d7d9-7e56-4f0d-85f8-d5449b803007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 20:06:05.181558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:05.173707 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb" Nov 03 20:06:05.305305 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:05.300458 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d40208cf-757c-4b13-a2f6-722e534b1f63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d40208cf-757c-4b13-a2f6-722e534b1f63-maas-api-openshift-ca\") pod \"d40208cf-757c-4b13-a2f6-722e534b1f63\" (UID: \"d40208cf-757c-4b13-a2f6-722e534b1f63\") " Nov 03 20:06:05.305305 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:05.300752 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40208cf-757c-4b13-a2f6-722e534b1f63-kube-api-access-xj2mx\" (UniqueName: \"kubernetes.io/projected/d40208cf-757c-4b13-a2f6-722e534b1f63-kube-api-access-xj2mx\") pod \"d40208cf-757c-4b13-a2f6-722e534b1f63\" (UID: \"d40208cf-757c-4b13-a2f6-722e534b1f63\") " Nov 03 20:06:05.305305 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:05.300872 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d40208cf-757c-4b13-a2f6-722e534b1f63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d40208cf-757c-4b13-a2f6-722e534b1f63-maas-api-xks-ca\") pod \"d40208cf-757c-4b13-a2f6-722e534b1f63\" (UID: \"d40208cf-757c-4b13-a2f6-722e534b1f63\") " Nov 03 20:06:05.330271 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:05.330227 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d40208cf-757c-4b13-a2f6-722e534b1f63-maas-api-openshift-ca" pod "d40208cf-757c-4b13-a2f6-722e534b1f63" (UID: "d40208cf-757c-4b13-a2f6-722e534b1f63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:05.336261 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:06:05.336209 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:06:05.432004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:05.431960 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40208cf-757c-4b13-a2f6-722e534b1f63-kube-api-access-xj2mx" pod "d40208cf-757c-4b13-a2f6-722e534b1f63" (UID: "d40208cf-757c-4b13-a2f6-722e534b1f63"). InnerVolumeSpecName "kube-api-access-xj2mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:05.467626 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:05.439713 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d40208cf-757c-4b13-a2f6-722e534b1f63-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:05.467846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:05.462631 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:05.879756 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:05.869573 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj2mx\" (UniqueName: \"kubernetes.io/projected/d40208cf-757c-4b13-a2f6-722e534b1f63-kube-api-access-xj2mx\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:05.879756 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:05.869652 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:05.879756 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:05.870284 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:06:05.956313 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:05.956215 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 20:06:05.957618 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:05.957580 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" podUID="b205487d-c7cb-4491-a6e5-c90771872d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 20:06:06.035512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.034748 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:06.037754 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.036209 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:06:06.039842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.039228 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:06.043584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.040232 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:06:06.110890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.110853 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:06:06.111176 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.111143 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" podUID="87cff690-9d53-44d6-a1dc-3f96238ca235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 20:06:06.214864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.144021 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:06:06.232349 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.232292 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" podUID="75b8d7d9-7e56-4f0d-85f8-d5449b803007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 20:06:06.232574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.174897 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb" Nov 03 20:06:06.239944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.239866 5111 generic.go:356] "Generic (PLEG): container finished" podID="60956010-1d2b-4873-bff8-384c42f66e4d" containerID="0d9a5ab2b49abfb88fda7efe2a7a9acd88af8bc1980067a3de96382173ad0cb2" exitCode=0 Nov 03 20:06:06.241619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.174554 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb" event={"ID":"d40208cf-757c-4b13-a2f6-722e534b1f63","Type":"ContainerDied","Data":"1149b64e1d5456310a5ffcb28e5574db88ea4e309774798db9d1089cb8f6183d"} Nov 03 20:06:06.241742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.241683 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1149b64e1d5456310a5ffcb28e5574db88ea4e309774798db9d1089cb8f6183d" Nov 03 20:06:06.241742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.241725 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz" event={"ID":"60956010-1d2b-4873-bff8-384c42f66e4d","Type":"ContainerDied","Data":"0d9a5ab2b49abfb88fda7efe2a7a9acd88af8bc1980067a3de96382173ad0cb2"} Nov 03 20:06:06.363952 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.356628 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d40208cf-757c-4b13-a2f6-722e534b1f63-maas-api-xks-ca" pod "d40208cf-757c-4b13-a2f6-722e534b1f63" (UID: "d40208cf-757c-4b13-a2f6-722e534b1f63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:06.363952 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.361293 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq"] Nov 03 20:06:06.421578 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.421545 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d40208cf-757c-4b13-a2f6-722e534b1f63-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:06.441344 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.440624 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:06:06.442925 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.442685 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:06:06.594602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.591738 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 20:06:06.594602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.591795 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" podUID="b205487d-c7cb-4491-a6e5-c90771872d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 20:06:06.679559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.675380 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:06:06.679559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.675440 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" podUID="87cff690-9d53-44d6-a1dc-3f96238ca235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 20:06:06.994912 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.994872 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6"] Nov 03 20:06:06.995758 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.995735 5111 cpu_manager.go:414] "containerMap: removing container" podUID="d40208cf-757c-4b13-a2f6-722e534b1f63" containerName="revoke-keys" Nov 03 20:06:06.995758 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.995757 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d40208cf-757c-4b13-a2f6-722e534b1f63" containerName="revoke-keys" Nov 03 20:06:06.995933 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:06.995909 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="d40208cf-757c-4b13-a2f6-722e534b1f63" containerName="revoke-keys" Nov 03 20:06:07.016556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.004762 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6" Nov 03 20:06:07.033685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.033655 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6"] Nov 03 20:06:07.035725 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.035090 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njztd\" (UniqueName: \"kubernetes.io/projected/4b1849d3-b100-41a1-b92a-df99efc5657f-kube-api-access-njztd\") pod \"maas-api-revoke-sub-8a1028db-g9ch6\" (UID: \"4b1849d3-b100-41a1-b92a-df99efc5657f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6" Nov 03 20:06:07.036091 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.036058 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1849d3-b100-41a1-b92a-df99efc5657f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a1028db-g9ch6\" (UID: \"4b1849d3-b100-41a1-b92a-df99efc5657f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6" Nov 03 20:06:07.036418 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.036397 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1849d3-b100-41a1-b92a-df99efc5657f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a1028db-g9ch6\" (UID: \"4b1849d3-b100-41a1-b92a-df99efc5657f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6" Nov 03 20:06:07.109788 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.109362 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:06:07.109788 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.109418 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" podUID="75b8d7d9-7e56-4f0d-85f8-d5449b803007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 20:06:07.143797 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.140915 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1849d3-b100-41a1-b92a-df99efc5657f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a1028db-g9ch6\" (UID: \"4b1849d3-b100-41a1-b92a-df99efc5657f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6" Nov 03 20:06:07.143797 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.141524 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1849d3-b100-41a1-b92a-df99efc5657f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a1028db-g9ch6\" (UID: \"4b1849d3-b100-41a1-b92a-df99efc5657f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6" Nov 03 20:06:07.143797 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.141793 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njztd\" (UniqueName: \"kubernetes.io/projected/4b1849d3-b100-41a1-b92a-df99efc5657f-kube-api-access-njztd\") pod \"maas-api-revoke-sub-8a1028db-g9ch6\" (UID: \"4b1849d3-b100-41a1-b92a-df99efc5657f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6" Nov 03 20:06:07.143797 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.142929 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1849d3-b100-41a1-b92a-df99efc5657f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a1028db-g9ch6\" (UID: \"4b1849d3-b100-41a1-b92a-df99efc5657f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6" Nov 03 20:06:07.195580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.195059 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1849d3-b100-41a1-b92a-df99efc5657f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a1028db-g9ch6\" (UID: \"4b1849d3-b100-41a1-b92a-df99efc5657f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6" Nov 03 20:06:07.197765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.197684 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njztd\" (UniqueName: \"kubernetes.io/projected/4b1849d3-b100-41a1-b92a-df99efc5657f-kube-api-access-njztd\") pod \"maas-api-revoke-sub-8a1028db-g9ch6\" (UID: \"4b1849d3-b100-41a1-b92a-df99efc5657f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6" Nov 03 20:06:07.283838 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.283751 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" event={"ID":"2050bea5-d734-487f-ba88-5eb6b4d67f81","Type":"ContainerStarted","Data":"6e4b021992152632fbfcdf755b2ba00f3ea62347438ae1e76ae570f88c76a69f"} Nov 03 20:06:07.283838 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.283807 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" event={"ID":"2050bea5-d734-487f-ba88-5eb6b4d67f81","Type":"ContainerStarted","Data":"bb82b0f9b76ccad7197b10d4cfd9aa9bc844de63b95cef3474cb8ad155974f4e"} Nov 03 20:06:07.354247 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.351301 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" podStartSLOduration=5.351280448 podStartE2EDuration="5.351280448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:07.336984345 +0000 UTC m=+1597.299362784" watchObservedRunningTime="2026-11-03 20:06:07.351280448 +0000 UTC m=+1597.313658888" Nov 03 20:06:07.418292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.417367 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6" Nov 03 20:06:07.576618 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.575623 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:06:07.577800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.577349 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 20:06:07.577800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.577400 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" podUID="b205487d-c7cb-4491-a6e5-c90771872d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 20:06:07.703263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.702727 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:06:07.732494 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.732110 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz" Nov 03 20:06:07.838975 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.838893 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60956010-1d2b-4873-bff8-384c42f66e4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60956010-1d2b-4873-bff8-384c42f66e4d-maas-api-openshift-ca\") pod \"60956010-1d2b-4873-bff8-384c42f66e4d\" (UID: \"60956010-1d2b-4873-bff8-384c42f66e4d\") " Nov 03 20:06:07.839142 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.839023 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60956010-1d2b-4873-bff8-384c42f66e4d-kube-api-access-8zspc\" (UniqueName: \"kubernetes.io/projected/60956010-1d2b-4873-bff8-384c42f66e4d-kube-api-access-8zspc\") pod \"60956010-1d2b-4873-bff8-384c42f66e4d\" (UID: \"60956010-1d2b-4873-bff8-384c42f66e4d\") " Nov 03 20:06:07.839210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.839192 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60956010-1d2b-4873-bff8-384c42f66e4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60956010-1d2b-4873-bff8-384c42f66e4d-maas-api-xks-ca\") pod \"60956010-1d2b-4873-bff8-384c42f66e4d\" (UID: \"60956010-1d2b-4873-bff8-384c42f66e4d\") " Nov 03 20:06:07.840496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.840471 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60956010-1d2b-4873-bff8-384c42f66e4d-maas-api-openshift-ca" pod "60956010-1d2b-4873-bff8-384c42f66e4d" (UID: "60956010-1d2b-4873-bff8-384c42f66e4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:07.842751 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.842726 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60956010-1d2b-4873-bff8-384c42f66e4d-maas-api-xks-ca" pod "60956010-1d2b-4873-bff8-384c42f66e4d" (UID: "60956010-1d2b-4873-bff8-384c42f66e4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:07.844094 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.844062 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60956010-1d2b-4873-bff8-384c42f66e4d-kube-api-access-8zspc" pod "60956010-1d2b-4873-bff8-384c42f66e4d" (UID: "60956010-1d2b-4873-bff8-384c42f66e4d"). InnerVolumeSpecName "kube-api-access-8zspc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:07.942316 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.942278 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60956010-1d2b-4873-bff8-384c42f66e4d-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:07.942635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.942610 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zspc\" (UniqueName: \"kubernetes.io/projected/60956010-1d2b-4873-bff8-384c42f66e4d-kube-api-access-8zspc\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:07.942787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.942772 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60956010-1d2b-4873-bff8-384c42f66e4d-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:07.943562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:07.943493 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6"] Nov 03 20:06:08.074065 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:08.073656 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:06:08.104605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:08.104484 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:06:08.104605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:08.104573 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" podUID="75b8d7d9-7e56-4f0d-85f8-d5449b803007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 20:06:08.298497 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:08.298459 5111 generic.go:356] "Generic (PLEG): container finished" podID="2050bea5-d734-487f-ba88-5eb6b4d67f81" containerID="6e4b021992152632fbfcdf755b2ba00f3ea62347438ae1e76ae570f88c76a69f" exitCode=0 Nov 03 20:06:08.298706 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:08.298555 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" event={"ID":"2050bea5-d734-487f-ba88-5eb6b4d67f81","Type":"ContainerDied","Data":"6e4b021992152632fbfcdf755b2ba00f3ea62347438ae1e76ae570f88c76a69f"} Nov 03 20:06:08.300339 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:08.300316 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz" Nov 03 20:06:08.300455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:08.300360 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz" event={"ID":"60956010-1d2b-4873-bff8-384c42f66e4d","Type":"ContainerDied","Data":"7187ef42bb9a915950a830f94f3d80105f3544222c11bb874972b9cc907ff3b4"} Nov 03 20:06:08.300455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:08.300403 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7187ef42bb9a915950a830f94f3d80105f3544222c11bb874972b9cc907ff3b4" Nov 03 20:06:08.302125 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:08.302085 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6" event={"ID":"4b1849d3-b100-41a1-b92a-df99efc5657f","Type":"ContainerStarted","Data":"97b3c8e07a756c427a54a65ee7e402da0eb4aee66032d4ce0a79a4d39e5ac640"} Nov 03 20:06:08.303824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:08.303805 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:06:08.566633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:08.566601 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:06:08.567918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:08.567894 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:06:09.104544 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.104483 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:06:09.105071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.104819 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:06:09.105071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.104877 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" podUID="75b8d7d9-7e56-4f0d-85f8-d5449b803007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 20:06:09.306212 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.306179 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d"] Nov 03 20:06:09.307317 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.307288 5111 cpu_manager.go:414] "containerMap: removing container" podUID="60956010-1d2b-4873-bff8-384c42f66e4d" containerName="revoke-keys" Nov 03 20:06:09.307317 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.307315 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60956010-1d2b-4873-bff8-384c42f66e4d" containerName="revoke-keys" Nov 03 20:06:09.307632 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.307608 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="60956010-1d2b-4873-bff8-384c42f66e4d" containerName="revoke-keys" Nov 03 20:06:09.311775 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.311755 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d" Nov 03 20:06:09.316591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.316479 5111 generic.go:356] "Generic (PLEG): container finished" podID="4b1849d3-b100-41a1-b92a-df99efc5657f" containerID="d231b7b03c619efc3960516ffae050b6ab5363bf65a2edd491159be1af8c4903" exitCode=0 Nov 03 20:06:09.317297 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.317272 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6" event={"ID":"4b1849d3-b100-41a1-b92a-df99efc5657f","Type":"ContainerDied","Data":"d231b7b03c619efc3960516ffae050b6ab5363bf65a2edd491159be1af8c4903"} Nov 03 20:06:09.322890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.322866 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d"] Nov 03 20:06:09.376505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.376465 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35443837-swl8d\" (UID: \"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d" Nov 03 20:06:09.376760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.376731 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35443837-swl8d\" (UID: \"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d" Nov 03 20:06:09.376850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.376784 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwgwn\" (UniqueName: \"kubernetes.io/projected/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-kube-api-access-cwgwn\") pod \"maas-api-revoke-sub-35443837-swl8d\" (UID: \"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d" Nov 03 20:06:09.481549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.479962 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35443837-swl8d\" (UID: \"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d" Nov 03 20:06:09.481549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.480355 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35443837-swl8d\" (UID: \"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d" Nov 03 20:06:09.489883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.480789 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwgwn\" (UniqueName: \"kubernetes.io/projected/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-kube-api-access-cwgwn\") pod \"maas-api-revoke-sub-35443837-swl8d\" (UID: \"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d" Nov 03 20:06:09.489883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.485247 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35443837-swl8d\" (UID: \"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d" Nov 03 20:06:09.489883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.489730 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35443837-swl8d\" (UID: \"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d" Nov 03 20:06:09.498572 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.498502 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwgwn\" (UniqueName: \"kubernetes.io/projected/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-kube-api-access-cwgwn\") pod \"maas-api-revoke-sub-35443837-swl8d\" (UID: \"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d" Nov 03 20:06:09.582223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.582145 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" Nov 03 20:06:09.636883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.636849 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d" Nov 03 20:06:09.684456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.684422 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2050bea5-d734-487f-ba88-5eb6b4d67f81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2050bea5-d734-487f-ba88-5eb6b4d67f81-maas-api-openshift-ca\") pod \"2050bea5-d734-487f-ba88-5eb6b4d67f81\" (UID: \"2050bea5-d734-487f-ba88-5eb6b4d67f81\") " Nov 03 20:06:09.684632 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.684602 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2050bea5-d734-487f-ba88-5eb6b4d67f81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2050bea5-d734-487f-ba88-5eb6b4d67f81-maas-api-xks-ca\") pod \"2050bea5-d734-487f-ba88-5eb6b4d67f81\" (UID: \"2050bea5-d734-487f-ba88-5eb6b4d67f81\") " Nov 03 20:06:09.684767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.684752 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2050bea5-d734-487f-ba88-5eb6b4d67f81-kube-api-access-4lq9v\" (UniqueName: \"kubernetes.io/projected/2050bea5-d734-487f-ba88-5eb6b4d67f81-kube-api-access-4lq9v\") pod \"2050bea5-d734-487f-ba88-5eb6b4d67f81\" (UID: \"2050bea5-d734-487f-ba88-5eb6b4d67f81\") " Nov 03 20:06:09.685206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.685069 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2050bea5-d734-487f-ba88-5eb6b4d67f81-maas-api-openshift-ca" pod "2050bea5-d734-487f-ba88-5eb6b4d67f81" (UID: "2050bea5-d734-487f-ba88-5eb6b4d67f81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:09.685516 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.685497 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2050bea5-d734-487f-ba88-5eb6b4d67f81-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:09.686502 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.686482 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2050bea5-d734-487f-ba88-5eb6b4d67f81-maas-api-xks-ca" pod "2050bea5-d734-487f-ba88-5eb6b4d67f81" (UID: "2050bea5-d734-487f-ba88-5eb6b4d67f81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:09.687072 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.687028 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2050bea5-d734-487f-ba88-5eb6b4d67f81-kube-api-access-4lq9v" pod "2050bea5-d734-487f-ba88-5eb6b4d67f81" (UID: "2050bea5-d734-487f-ba88-5eb6b4d67f81"). InnerVolumeSpecName "kube-api-access-4lq9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:09.787820 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.787784 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lq9v\" (UniqueName: \"kubernetes.io/projected/2050bea5-d734-487f-ba88-5eb6b4d67f81-kube-api-access-4lq9v\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:09.787820 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.787816 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2050bea5-d734-487f-ba88-5eb6b4d67f81-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:09.960371 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:06:09.960323 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7ac9a4_aa54_4eed_a40f_fb0d77a6ee4f.slice/crio-5a4287dae3afd4bca9192e089d6a27fc5de6ee389d45ccd5e6efd0d6e8abb004 WatchSource:0}: Error finding container 5a4287dae3afd4bca9192e089d6a27fc5de6ee389d45ccd5e6efd0d6e8abb004: Status 404 returned error can't find the container with id 5a4287dae3afd4bca9192e089d6a27fc5de6ee389d45ccd5e6efd0d6e8abb004 Nov 03 20:06:09.967029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:09.966990 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d"] Nov 03 20:06:10.109014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:10.108987 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:06:10.110041 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:10.110020 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:06:10.346250 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:10.346212 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" event={"ID":"2050bea5-d734-487f-ba88-5eb6b4d67f81","Type":"ContainerDied","Data":"bb82b0f9b76ccad7197b10d4cfd9aa9bc844de63b95cef3474cb8ad155974f4e"} Nov 03 20:06:10.346523 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:10.346504 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb82b0f9b76ccad7197b10d4cfd9aa9bc844de63b95cef3474cb8ad155974f4e" Nov 03 20:06:10.346645 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:10.346266 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq" Nov 03 20:06:10.355478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:10.355443 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d" event={"ID":"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f","Type":"ContainerStarted","Data":"5a4287dae3afd4bca9192e089d6a27fc5de6ee389d45ccd5e6efd0d6e8abb004"} Nov 03 20:06:10.554631 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:10.554607 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6" Nov 03 20:06:10.604738 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:10.604702 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1849d3-b100-41a1-b92a-df99efc5657f-kube-api-access-njztd\" (UniqueName: \"kubernetes.io/projected/4b1849d3-b100-41a1-b92a-df99efc5657f-kube-api-access-njztd\") pod \"4b1849d3-b100-41a1-b92a-df99efc5657f\" (UID: \"4b1849d3-b100-41a1-b92a-df99efc5657f\") " Nov 03 20:06:10.604993 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:10.604967 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b1849d3-b100-41a1-b92a-df99efc5657f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1849d3-b100-41a1-b92a-df99efc5657f-maas-api-xks-ca\") pod \"4b1849d3-b100-41a1-b92a-df99efc5657f\" (UID: \"4b1849d3-b100-41a1-b92a-df99efc5657f\") " Nov 03 20:06:10.605205 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:10.605179 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b1849d3-b100-41a1-b92a-df99efc5657f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1849d3-b100-41a1-b92a-df99efc5657f-maas-api-openshift-ca\") pod \"4b1849d3-b100-41a1-b92a-df99efc5657f\" (UID: \"4b1849d3-b100-41a1-b92a-df99efc5657f\") " Nov 03 20:06:10.605587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:10.605512 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1849d3-b100-41a1-b92a-df99efc5657f-maas-api-openshift-ca" pod "4b1849d3-b100-41a1-b92a-df99efc5657f" (UID: "4b1849d3-b100-41a1-b92a-df99efc5657f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:10.605962 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:10.605937 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1849d3-b100-41a1-b92a-df99efc5657f-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:10.607224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:10.607201 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1849d3-b100-41a1-b92a-df99efc5657f-maas-api-xks-ca" pod "4b1849d3-b100-41a1-b92a-df99efc5657f" (UID: "4b1849d3-b100-41a1-b92a-df99efc5657f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:10.607313 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:10.607259 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1849d3-b100-41a1-b92a-df99efc5657f-kube-api-access-njztd" pod "4b1849d3-b100-41a1-b92a-df99efc5657f" (UID: "4b1849d3-b100-41a1-b92a-df99efc5657f"). InnerVolumeSpecName "kube-api-access-njztd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:10.707145 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:10.707112 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1849d3-b100-41a1-b92a-df99efc5657f-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:10.707145 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:10.707141 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njztd\" (UniqueName: \"kubernetes.io/projected/4b1849d3-b100-41a1-b92a-df99efc5657f-kube-api-access-njztd\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:11.366784 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:11.366695 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6" Nov 03 20:06:11.366784 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:11.366736 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6" event={"ID":"4b1849d3-b100-41a1-b92a-df99efc5657f","Type":"ContainerDied","Data":"97b3c8e07a756c427a54a65ee7e402da0eb4aee66032d4ce0a79a4d39e5ac640"} Nov 03 20:06:11.366784 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:11.366782 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97b3c8e07a756c427a54a65ee7e402da0eb4aee66032d4ce0a79a4d39e5ac640" Nov 03 20:06:11.368696 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:11.368666 5111 generic.go:356] "Generic (PLEG): container finished" podID="6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f" containerID="5a714145ee047a6c52e00b2dc71bb5271a72c082324bd7419e4a19d36dca5887" exitCode=0 Nov 03 20:06:11.368826 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:11.368715 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d" event={"ID":"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f","Type":"ContainerDied","Data":"5a714145ee047a6c52e00b2dc71bb5271a72c082324bd7419e4a19d36dca5887"} Nov 03 20:06:12.569457 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:12.569429 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d" Nov 03 20:06:12.643424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:12.643344 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-maas-api-xks-ca\") pod \"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f\" (UID: \"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f\") " Nov 03 20:06:12.643600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:12.643546 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-maas-api-openshift-ca\") pod \"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f\" (UID: \"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f\") " Nov 03 20:06:12.643600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:12.643575 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-kube-api-access-cwgwn\" (UniqueName: \"kubernetes.io/projected/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-kube-api-access-cwgwn\") pod \"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f\" (UID: \"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f\") " Nov 03 20:06:12.644014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:12.643959 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-maas-api-openshift-ca" pod "6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f" (UID: "6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:12.644231 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:12.644205 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:12.646222 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:12.646192 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-maas-api-xks-ca" pod "6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f" (UID: "6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:12.646395 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:12.646373 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-kube-api-access-cwgwn" pod "6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f" (UID: "6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f"). InnerVolumeSpecName "kube-api-access-cwgwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:12.745391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:12.745359 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwgwn\" (UniqueName: \"kubernetes.io/projected/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-kube-api-access-cwgwn\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:12.745391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:12.745390 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:13.391900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.391858 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d" event={"ID":"6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f","Type":"ContainerDied","Data":"5a4287dae3afd4bca9192e089d6a27fc5de6ee389d45ccd5e6efd0d6e8abb004"} Nov 03 20:06:13.392096 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.391930 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a4287dae3afd4bca9192e089d6a27fc5de6ee389d45ccd5e6efd0d6e8abb004" Nov 03 20:06:13.392096 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.392006 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d" Nov 03 20:06:13.506205 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.506169 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2"] Nov 03 20:06:13.507225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.507198 5111 cpu_manager.go:414] "containerMap: removing container" podUID="2050bea5-d734-487f-ba88-5eb6b4d67f81" containerName="revoke-keys" Nov 03 20:06:13.507225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.507223 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2050bea5-d734-487f-ba88-5eb6b4d67f81" containerName="revoke-keys" Nov 03 20:06:13.507421 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.507252 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f" containerName="revoke-keys" Nov 03 20:06:13.507421 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.507258 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f" containerName="revoke-keys" Nov 03 20:06:13.507421 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.507290 5111 cpu_manager.go:414] "containerMap: removing container" podUID="4b1849d3-b100-41a1-b92a-df99efc5657f" containerName="revoke-keys" Nov 03 20:06:13.507421 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.507298 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1849d3-b100-41a1-b92a-df99efc5657f" containerName="revoke-keys" Nov 03 20:06:13.507640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.507467 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f" containerName="revoke-keys" Nov 03 20:06:13.507640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.507496 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1849d3-b100-41a1-b92a-df99efc5657f" containerName="revoke-keys" Nov 03 20:06:13.507640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.507511 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="2050bea5-d734-487f-ba88-5eb6b4d67f81" containerName="revoke-keys" Nov 03 20:06:13.511915 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.511891 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2" Nov 03 20:06:13.518653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.518598 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:06:13.518760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.518682 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:06:13.518760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.518681 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:06:13.518760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.518694 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:06:13.527050 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.526460 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2"] Nov 03 20:06:13.666239 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.666159 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb4377fe-0214-4470-9b01-f7b42f268f27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43b392f6-lqqd2\" (UID: \"cb4377fe-0214-4470-9b01-f7b42f268f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2" Nov 03 20:06:13.666671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.666319 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmq4z\" (UniqueName: \"kubernetes.io/projected/cb4377fe-0214-4470-9b01-f7b42f268f27-kube-api-access-gmq4z\") pod \"maas-api-revoke-sub-43b392f6-lqqd2\" (UID: \"cb4377fe-0214-4470-9b01-f7b42f268f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2" Nov 03 20:06:13.666671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.666353 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb4377fe-0214-4470-9b01-f7b42f268f27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43b392f6-lqqd2\" (UID: \"cb4377fe-0214-4470-9b01-f7b42f268f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2" Nov 03 20:06:13.767508 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.767471 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb4377fe-0214-4470-9b01-f7b42f268f27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43b392f6-lqqd2\" (UID: \"cb4377fe-0214-4470-9b01-f7b42f268f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2" Nov 03 20:06:13.767694 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.767676 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmq4z\" (UniqueName: \"kubernetes.io/projected/cb4377fe-0214-4470-9b01-f7b42f268f27-kube-api-access-gmq4z\") pod \"maas-api-revoke-sub-43b392f6-lqqd2\" (UID: \"cb4377fe-0214-4470-9b01-f7b42f268f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2" Nov 03 20:06:13.767737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.767722 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb4377fe-0214-4470-9b01-f7b42f268f27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43b392f6-lqqd2\" (UID: \"cb4377fe-0214-4470-9b01-f7b42f268f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2" Nov 03 20:06:13.768583 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.768559 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb4377fe-0214-4470-9b01-f7b42f268f27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43b392f6-lqqd2\" (UID: \"cb4377fe-0214-4470-9b01-f7b42f268f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2" Nov 03 20:06:13.770259 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.770239 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb4377fe-0214-4470-9b01-f7b42f268f27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43b392f6-lqqd2\" (UID: \"cb4377fe-0214-4470-9b01-f7b42f268f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2" Nov 03 20:06:13.780643 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.780617 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmq4z\" (UniqueName: \"kubernetes.io/projected/cb4377fe-0214-4470-9b01-f7b42f268f27-kube-api-access-gmq4z\") pod \"maas-api-revoke-sub-43b392f6-lqqd2\" (UID: \"cb4377fe-0214-4470-9b01-f7b42f268f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2" Nov 03 20:06:13.832190 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:13.832164 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2" Nov 03 20:06:14.150979 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:06:14.150943 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4377fe_0214_4470_9b01_f7b42f268f27.slice/crio-dd8aaa43b86bc440db0cf0613649f382cc99c0483b197bc26a393b0bbee7a6fb WatchSource:0}: Error finding container dd8aaa43b86bc440db0cf0613649f382cc99c0483b197bc26a393b0bbee7a6fb: Status 404 returned error can't find the container with id dd8aaa43b86bc440db0cf0613649f382cc99c0483b197bc26a393b0bbee7a6fb Nov 03 20:06:14.152544 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:14.152506 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2"] Nov 03 20:06:14.403180 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:14.403099 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2" event={"ID":"cb4377fe-0214-4470-9b01-f7b42f268f27","Type":"ContainerStarted","Data":"dd8aaa43b86bc440db0cf0613649f382cc99c0483b197bc26a393b0bbee7a6fb"} Nov 03 20:06:14.972298 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:14.972217 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f"] Nov 03 20:06:14.977057 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:14.977038 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" Nov 03 20:06:14.989949 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:14.989344 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f"] Nov 03 20:06:15.090295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:15.090260 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e145ff2-89d8-45f2-b777-a78e17636e47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bc1749b-jgq6f\" (UID: \"7e145ff2-89d8-45f2-b777-a78e17636e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" Nov 03 20:06:15.090459 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:15.090374 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e145ff2-89d8-45f2-b777-a78e17636e47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bc1749b-jgq6f\" (UID: \"7e145ff2-89d8-45f2-b777-a78e17636e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" Nov 03 20:06:15.090508 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:15.090448 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l7m2\" (UniqueName: \"kubernetes.io/projected/7e145ff2-89d8-45f2-b777-a78e17636e47-kube-api-access-4l7m2\") pod \"maas-api-revoke-sub-0bc1749b-jgq6f\" (UID: \"7e145ff2-89d8-45f2-b777-a78e17636e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" Nov 03 20:06:15.191997 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:15.191957 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e145ff2-89d8-45f2-b777-a78e17636e47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bc1749b-jgq6f\" (UID: \"7e145ff2-89d8-45f2-b777-a78e17636e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" Nov 03 20:06:15.192178 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:15.192072 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e145ff2-89d8-45f2-b777-a78e17636e47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bc1749b-jgq6f\" (UID: \"7e145ff2-89d8-45f2-b777-a78e17636e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" Nov 03 20:06:15.192178 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:15.192125 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l7m2\" (UniqueName: \"kubernetes.io/projected/7e145ff2-89d8-45f2-b777-a78e17636e47-kube-api-access-4l7m2\") pod \"maas-api-revoke-sub-0bc1749b-jgq6f\" (UID: \"7e145ff2-89d8-45f2-b777-a78e17636e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" Nov 03 20:06:15.192932 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:15.192909 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e145ff2-89d8-45f2-b777-a78e17636e47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bc1749b-jgq6f\" (UID: \"7e145ff2-89d8-45f2-b777-a78e17636e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" Nov 03 20:06:15.194461 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:15.194436 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e145ff2-89d8-45f2-b777-a78e17636e47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bc1749b-jgq6f\" (UID: \"7e145ff2-89d8-45f2-b777-a78e17636e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" Nov 03 20:06:15.201524 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:15.201501 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l7m2\" (UniqueName: \"kubernetes.io/projected/7e145ff2-89d8-45f2-b777-a78e17636e47-kube-api-access-4l7m2\") pod \"maas-api-revoke-sub-0bc1749b-jgq6f\" (UID: \"7e145ff2-89d8-45f2-b777-a78e17636e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" Nov 03 20:06:15.297702 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:15.297662 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" Nov 03 20:06:15.415778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:15.415740 5111 generic.go:356] "Generic (PLEG): container finished" podID="cb4377fe-0214-4470-9b01-f7b42f268f27" containerID="f342039b462ce05ed70473e4b7667ed8ce011d812dc69fc8e2a6f4c91a6aa092" exitCode=0 Nov 03 20:06:15.415945 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:15.415822 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2" event={"ID":"cb4377fe-0214-4470-9b01-f7b42f268f27","Type":"ContainerDied","Data":"f342039b462ce05ed70473e4b7667ed8ce011d812dc69fc8e2a6f4c91a6aa092"} Nov 03 20:06:15.607616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:15.607589 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f"] Nov 03 20:06:16.065887 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.065859 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb"] Nov 03 20:06:16.070901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.070880 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" Nov 03 20:06:16.097038 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.096720 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb"] Nov 03 20:06:16.213884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.213757 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dt9l\" (UniqueName: \"kubernetes.io/projected/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-kube-api-access-2dt9l\") pod \"maas-api-revoke-sub-48d573db-qx4vb\" (UID: \"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" Nov 03 20:06:16.214357 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.214326 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48d573db-qx4vb\" (UID: \"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" Nov 03 20:06:16.214575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.214462 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48d573db-qx4vb\" (UID: \"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" Nov 03 20:06:16.316802 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.316767 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dt9l\" (UniqueName: \"kubernetes.io/projected/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-kube-api-access-2dt9l\") pod \"maas-api-revoke-sub-48d573db-qx4vb\" (UID: \"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" Nov 03 20:06:16.316998 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.316872 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48d573db-qx4vb\" (UID: \"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" Nov 03 20:06:16.317068 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.316984 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48d573db-qx4vb\" (UID: \"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" Nov 03 20:06:16.317574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.317549 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48d573db-qx4vb\" (UID: \"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" Nov 03 20:06:16.319635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.319615 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48d573db-qx4vb\" (UID: \"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" Nov 03 20:06:16.326493 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.326473 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dt9l\" (UniqueName: \"kubernetes.io/projected/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-kube-api-access-2dt9l\") pod \"maas-api-revoke-sub-48d573db-qx4vb\" (UID: \"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" Nov 03 20:06:16.419091 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.418806 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" Nov 03 20:06:16.610565 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.602142 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" event={"ID":"7e145ff2-89d8-45f2-b777-a78e17636e47","Type":"ContainerStarted","Data":"a14e8538fc6cc000b7242e144102cdb4f9c0acf8332593449dfa05ff473e135e"} Nov 03 20:06:16.610565 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.602237 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" event={"ID":"7e145ff2-89d8-45f2-b777-a78e17636e47","Type":"ContainerStarted","Data":"9aaa0f420a5e3115e5b35c8c0c065fe6fca8e3a668f17fb895f0fa347aa22d8a"} Nov 03 20:06:16.640715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.640565 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" podStartSLOduration=2.640544667 podStartE2EDuration="2.640544667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:16.629695479 +0000 UTC m=+1606.592073955" watchObservedRunningTime="2026-11-03 20:06:16.640544667 +0000 UTC m=+1606.602923104" Nov 03 20:06:16.921777 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:16.921745 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2" Nov 03 20:06:17.034183 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:06:17.034151 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace0c2ed_b808_4d4c_bf6c_1910edbd3b0e.slice/crio-1fc9ad7827222530b98922270a0009fc117a88f75db3083033f795a851187188 WatchSource:0}: Error finding container 1fc9ad7827222530b98922270a0009fc117a88f75db3083033f795a851187188: Status 404 returned error can't find the container with id 1fc9ad7827222530b98922270a0009fc117a88f75db3083033f795a851187188 Nov 03 20:06:17.035795 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.035764 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb4377fe-0214-4470-9b01-f7b42f268f27-kube-api-access-gmq4z\" (UniqueName: \"kubernetes.io/projected/cb4377fe-0214-4470-9b01-f7b42f268f27-kube-api-access-gmq4z\") pod \"cb4377fe-0214-4470-9b01-f7b42f268f27\" (UID: \"cb4377fe-0214-4470-9b01-f7b42f268f27\") " Nov 03 20:06:17.036014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.035993 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb4377fe-0214-4470-9b01-f7b42f268f27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb4377fe-0214-4470-9b01-f7b42f268f27-maas-api-xks-ca\") pod \"cb4377fe-0214-4470-9b01-f7b42f268f27\" (UID: \"cb4377fe-0214-4470-9b01-f7b42f268f27\") " Nov 03 20:06:17.036316 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.036296 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb4377fe-0214-4470-9b01-f7b42f268f27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb4377fe-0214-4470-9b01-f7b42f268f27-maas-api-openshift-ca\") pod \"cb4377fe-0214-4470-9b01-f7b42f268f27\" (UID: \"cb4377fe-0214-4470-9b01-f7b42f268f27\") " Nov 03 20:06:17.037062 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.036907 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb4377fe-0214-4470-9b01-f7b42f268f27-maas-api-openshift-ca" pod "cb4377fe-0214-4470-9b01-f7b42f268f27" (UID: "cb4377fe-0214-4470-9b01-f7b42f268f27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:17.040318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.040197 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb4377fe-0214-4470-9b01-f7b42f268f27-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:17.041318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.041296 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4377fe-0214-4470-9b01-f7b42f268f27-kube-api-access-gmq4z" pod "cb4377fe-0214-4470-9b01-f7b42f268f27" (UID: "cb4377fe-0214-4470-9b01-f7b42f268f27"). InnerVolumeSpecName "kube-api-access-gmq4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:17.041892 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.041873 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb4377fe-0214-4470-9b01-f7b42f268f27-maas-api-xks-ca" pod "cb4377fe-0214-4470-9b01-f7b42f268f27" (UID: "cb4377fe-0214-4470-9b01-f7b42f268f27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:17.044261 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.044238 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb"] Nov 03 20:06:17.141973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.141894 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmq4z\" (UniqueName: \"kubernetes.io/projected/cb4377fe-0214-4470-9b01-f7b42f268f27-kube-api-access-gmq4z\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:17.141973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.141927 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb4377fe-0214-4470-9b01-f7b42f268f27-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:17.427701 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.427623 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw"] Nov 03 20:06:17.428401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.428381 5111 cpu_manager.go:414] "containerMap: removing container" podUID="cb4377fe-0214-4470-9b01-f7b42f268f27" containerName="revoke-keys" Nov 03 20:06:17.428401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.428399 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb4377fe-0214-4470-9b01-f7b42f268f27" containerName="revoke-keys" Nov 03 20:06:17.428711 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.428691 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb4377fe-0214-4470-9b01-f7b42f268f27" containerName="revoke-keys" Nov 03 20:06:17.434065 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.434044 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" Nov 03 20:06:17.451768 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.449033 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw"] Nov 03 20:06:17.549080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.549043 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7618a46-7bdzw\" (UID: \"615f29b1-a11e-4b5d-a89b-dfefb9b91c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" Nov 03 20:06:17.549298 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.549272 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd2wh\" (UniqueName: \"kubernetes.io/projected/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-kube-api-access-rd2wh\") pod \"maas-api-revoke-sub-a7618a46-7bdzw\" (UID: \"615f29b1-a11e-4b5d-a89b-dfefb9b91c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" Nov 03 20:06:17.549377 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.549310 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7618a46-7bdzw\" (UID: \"615f29b1-a11e-4b5d-a89b-dfefb9b91c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" Nov 03 20:06:17.612316 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.612286 5111 generic.go:356] "Generic (PLEG): container finished" podID="7e145ff2-89d8-45f2-b777-a78e17636e47" containerID="a14e8538fc6cc000b7242e144102cdb4f9c0acf8332593449dfa05ff473e135e" exitCode=0 Nov 03 20:06:17.612439 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.612332 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" event={"ID":"7e145ff2-89d8-45f2-b777-a78e17636e47","Type":"ContainerDied","Data":"a14e8538fc6cc000b7242e144102cdb4f9c0acf8332593449dfa05ff473e135e"} Nov 03 20:06:17.613853 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.613824 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2" Nov 03 20:06:17.613980 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.613870 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2" event={"ID":"cb4377fe-0214-4470-9b01-f7b42f268f27","Type":"ContainerDied","Data":"dd8aaa43b86bc440db0cf0613649f382cc99c0483b197bc26a393b0bbee7a6fb"} Nov 03 20:06:17.613980 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.613910 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd8aaa43b86bc440db0cf0613649f382cc99c0483b197bc26a393b0bbee7a6fb" Nov 03 20:06:17.615885 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.615857 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" event={"ID":"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e","Type":"ContainerStarted","Data":"15633b3c95a55849a44597dc0066188ca70fedcb43841c093ee882720349b5b3"} Nov 03 20:06:17.616055 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.616032 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" event={"ID":"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e","Type":"ContainerStarted","Data":"1fc9ad7827222530b98922270a0009fc117a88f75db3083033f795a851187188"} Nov 03 20:06:17.650841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.650809 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7618a46-7bdzw\" (UID: \"615f29b1-a11e-4b5d-a89b-dfefb9b91c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" Nov 03 20:06:17.651842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.651178 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd2wh\" (UniqueName: \"kubernetes.io/projected/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-kube-api-access-rd2wh\") pod \"maas-api-revoke-sub-a7618a46-7bdzw\" (UID: \"615f29b1-a11e-4b5d-a89b-dfefb9b91c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" Nov 03 20:06:17.651842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.651230 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7618a46-7bdzw\" (UID: \"615f29b1-a11e-4b5d-a89b-dfefb9b91c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" Nov 03 20:06:17.652027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.651866 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7618a46-7bdzw\" (UID: \"615f29b1-a11e-4b5d-a89b-dfefb9b91c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" Nov 03 20:06:17.654292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.654270 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7618a46-7bdzw\" (UID: \"615f29b1-a11e-4b5d-a89b-dfefb9b91c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" Nov 03 20:06:17.663357 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.663334 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd2wh\" (UniqueName: \"kubernetes.io/projected/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-kube-api-access-rd2wh\") pod \"maas-api-revoke-sub-a7618a46-7bdzw\" (UID: \"615f29b1-a11e-4b5d-a89b-dfefb9b91c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" Nov 03 20:06:17.679555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.679469 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" podStartSLOduration=1.679456065 podStartE2EDuration="1.679456065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:17.677995513 +0000 UTC m=+1607.640373952" watchObservedRunningTime="2026-11-03 20:06:17.679456065 +0000 UTC m=+1607.641834526" Nov 03 20:06:17.758125 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:17.758099 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" Nov 03 20:06:18.100425 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:18.099729 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw"] Nov 03 20:06:18.762169 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:18.762128 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" event={"ID":"615f29b1-a11e-4b5d-a89b-dfefb9b91c28","Type":"ContainerStarted","Data":"61666ee3fb78465588429543abae2dbd2d20b13b5abc502b489f4537f110fe12"} Nov 03 20:06:18.762739 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:18.762716 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" event={"ID":"615f29b1-a11e-4b5d-a89b-dfefb9b91c28","Type":"ContainerStarted","Data":"93774d808f02d27824a3a1764f5b091f98f6b516ee703db59a57037188b83f02"} Nov 03 20:06:18.768041 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:18.765980 5111 generic.go:356] "Generic (PLEG): container finished" podID="ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e" containerID="15633b3c95a55849a44597dc0066188ca70fedcb43841c093ee882720349b5b3" exitCode=0 Nov 03 20:06:18.768041 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:18.766062 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" event={"ID":"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e","Type":"ContainerDied","Data":"15633b3c95a55849a44597dc0066188ca70fedcb43841c093ee882720349b5b3"} Nov 03 20:06:18.816095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:18.814580 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" podStartSLOduration=1.8145573480000001 podStartE2EDuration="1.814557348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:18.809148441 +0000 UTC m=+1608.771526880" watchObservedRunningTime="2026-11-03 20:06:18.814557348 +0000 UTC m=+1608.776935792" Nov 03 20:06:19.019943 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.019913 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" Nov 03 20:06:19.100211 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.100173 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e145ff2-89d8-45f2-b777-a78e17636e47-kube-api-access-4l7m2\" (UniqueName: \"kubernetes.io/projected/7e145ff2-89d8-45f2-b777-a78e17636e47-kube-api-access-4l7m2\") pod \"7e145ff2-89d8-45f2-b777-a78e17636e47\" (UID: \"7e145ff2-89d8-45f2-b777-a78e17636e47\") " Nov 03 20:06:19.100403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.100254 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e145ff2-89d8-45f2-b777-a78e17636e47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e145ff2-89d8-45f2-b777-a78e17636e47-maas-api-xks-ca\") pod \"7e145ff2-89d8-45f2-b777-a78e17636e47\" (UID: \"7e145ff2-89d8-45f2-b777-a78e17636e47\") " Nov 03 20:06:19.100602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.100515 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e145ff2-89d8-45f2-b777-a78e17636e47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e145ff2-89d8-45f2-b777-a78e17636e47-maas-api-openshift-ca\") pod \"7e145ff2-89d8-45f2-b777-a78e17636e47\" (UID: \"7e145ff2-89d8-45f2-b777-a78e17636e47\") " Nov 03 20:06:19.100900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.100871 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e145ff2-89d8-45f2-b777-a78e17636e47-maas-api-openshift-ca" pod "7e145ff2-89d8-45f2-b777-a78e17636e47" (UID: "7e145ff2-89d8-45f2-b777-a78e17636e47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:19.101467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.101442 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e145ff2-89d8-45f2-b777-a78e17636e47-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:19.102387 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.102357 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e145ff2-89d8-45f2-b777-a78e17636e47-maas-api-xks-ca" pod "7e145ff2-89d8-45f2-b777-a78e17636e47" (UID: "7e145ff2-89d8-45f2-b777-a78e17636e47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:19.102663 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.102644 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e145ff2-89d8-45f2-b777-a78e17636e47-kube-api-access-4l7m2" pod "7e145ff2-89d8-45f2-b777-a78e17636e47" (UID: "7e145ff2-89d8-45f2-b777-a78e17636e47"). InnerVolumeSpecName "kube-api-access-4l7m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:19.203351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.203313 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e145ff2-89d8-45f2-b777-a78e17636e47-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:19.203351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.203347 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l7m2\" (UniqueName: \"kubernetes.io/projected/7e145ff2-89d8-45f2-b777-a78e17636e47-kube-api-access-4l7m2\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:19.252656 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.252564 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b"] Nov 03 20:06:19.253505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.253480 5111 cpu_manager.go:414] "containerMap: removing container" podUID="7e145ff2-89d8-45f2-b777-a78e17636e47" containerName="revoke-keys" Nov 03 20:06:19.253505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.253506 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e145ff2-89d8-45f2-b777-a78e17636e47" containerName="revoke-keys" Nov 03 20:06:19.253812 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.253797 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e145ff2-89d8-45f2-b777-a78e17636e47" containerName="revoke-keys" Nov 03 20:06:19.258480 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.258458 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.304446 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.304407 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-credential-socket\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.304630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.304591 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-workload-certs\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.304630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.304623 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-envoy\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.304782 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.304770 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-workload-socket\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.304824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.304804 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-podinfo\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.304960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.304937 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-token\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.305010 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.304998 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs56q\" (UniqueName: \"kubernetes.io/projected/746d6249-f875-423d-9efb-fa8e3b31fa63-kube-api-access-hs56q\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.305043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.305023 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746d6249-f875-423d-9efb-fa8e3b31fa63-istiod-ca-cert\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.305076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.305045 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-data\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.339719 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.339412 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b"] Nov 03 20:06:19.461715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.381063 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-1cca2dba-openshift-default-dockercfg-xssn9\"" Nov 03 20:06:19.461715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.409910 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-credential-socket\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.461715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.410030 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-workload-certs\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.461715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.410059 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-envoy\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.461715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.410180 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-workload-socket\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.461715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.410227 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-podinfo\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.461715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.410318 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-token\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.461715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.410367 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs56q\" (UniqueName: \"kubernetes.io/projected/746d6249-f875-423d-9efb-fa8e3b31fa63-kube-api-access-hs56q\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.461715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.410402 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746d6249-f875-423d-9efb-fa8e3b31fa63-istiod-ca-cert\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.461715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.410439 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-data\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.461715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.411250 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-credential-socket\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.557452 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.552113 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-workload-certs\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.557452 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.557279 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-envoy\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.573408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.573375 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-workload-socket\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.576757 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.576605 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-token\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.585498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.585455 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746d6249-f875-423d-9efb-fa8e3b31fa63-istiod-ca-cert\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.586199 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.586166 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-data\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.643290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.642463 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs56q\" (UniqueName: \"kubernetes.io/projected/746d6249-f875-423d-9efb-fa8e3b31fa63-kube-api-access-hs56q\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.643290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.642979 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-podinfo\") pod \"e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:19.826426 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.826335 5111 generic.go:356] "Generic (PLEG): container finished" podID="615f29b1-a11e-4b5d-a89b-dfefb9b91c28" containerID="61666ee3fb78465588429543abae2dbd2d20b13b5abc502b489f4537f110fe12" exitCode=0 Nov 03 20:06:19.827266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.827233 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" event={"ID":"615f29b1-a11e-4b5d-a89b-dfefb9b91c28","Type":"ContainerDied","Data":"61666ee3fb78465588429543abae2dbd2d20b13b5abc502b489f4537f110fe12"} Nov 03 20:06:19.839652 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.839620 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" Nov 03 20:06:19.841720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.839852 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f" event={"ID":"7e145ff2-89d8-45f2-b777-a78e17636e47","Type":"ContainerDied","Data":"9aaa0f420a5e3115e5b35c8c0c065fe6fca8e3a668f17fb895f0fa347aa22d8a"} Nov 03 20:06:19.841958 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.841912 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9aaa0f420a5e3115e5b35c8c0c065fe6fca8e3a668f17fb895f0fa347aa22d8a" Nov 03 20:06:19.894716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:19.894680 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:20.319577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.319526 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" Nov 03 20:06:20.478295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.478214 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-maas-api-openshift-ca\") pod \"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e\" (UID: \"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e\") " Nov 03 20:06:20.478484 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.478465 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-kube-api-access-2dt9l\" (UniqueName: \"kubernetes.io/projected/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-kube-api-access-2dt9l\") pod \"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e\" (UID: \"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e\") " Nov 03 20:06:20.478611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.478592 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-maas-api-xks-ca\") pod \"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e\" (UID: \"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e\") " Nov 03 20:06:20.478741 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.478595 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-maas-api-openshift-ca" pod "ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e" (UID: "ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:20.479167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.479145 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:20.480684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.480660 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-maas-api-xks-ca" pod "ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e" (UID: "ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:20.484913 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.484891 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-kube-api-access-2dt9l" pod "ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e" (UID: "ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e"). InnerVolumeSpecName "kube-api-access-2dt9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:20.493902 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.493881 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b"] Nov 03 20:06:20.496928 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:06:20.496905 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746d6249_f875_423d_9efb_fa8e3b31fa63.slice/crio-fe62c33236b91a239bd74b9264862fc547c5921d02fc46e7496330da6783a793 WatchSource:0}: Error finding container fe62c33236b91a239bd74b9264862fc547c5921d02fc46e7496330da6783a793: Status 404 returned error can't find the container with id fe62c33236b91a239bd74b9264862fc547c5921d02fc46e7496330da6783a793 Nov 03 20:06:20.499371 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.499339 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:06:20.499455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.499404 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:06:20.499455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.499435 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:06:20.580301 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.580275 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dt9l\" (UniqueName: \"kubernetes.io/projected/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-kube-api-access-2dt9l\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:20.580411 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.580307 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:20.855090 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.855000 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" event={"ID":"746d6249-f875-423d-9efb-fa8e3b31fa63","Type":"ContainerStarted","Data":"fccefc602fa3c84e1f5d4f64b2e7f2be2c28a66938a261dc53b50ec6c346fbfd"} Nov 03 20:06:20.855090 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.855061 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" event={"ID":"746d6249-f875-423d-9efb-fa8e3b31fa63","Type":"ContainerStarted","Data":"fe62c33236b91a239bd74b9264862fc547c5921d02fc46e7496330da6783a793"} Nov 03 20:06:20.857182 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.857151 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" Nov 03 20:06:20.857307 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.857189 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb" event={"ID":"ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e","Type":"ContainerDied","Data":"1fc9ad7827222530b98922270a0009fc117a88f75db3083033f795a851187188"} Nov 03 20:06:20.857307 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.857231 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fc9ad7827222530b98922270a0009fc117a88f75db3083033f795a851187188" Nov 03 20:06:20.890446 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.889607 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" podStartSLOduration=1.889592956 podStartE2EDuration="1.889592956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:20.884215611 +0000 UTC m=+1610.846594055" watchObservedRunningTime="2026-11-03 20:06:20.889592956 +0000 UTC m=+1610.851971395" Nov 03 20:06:20.895410 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:20.895337 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:21.074299 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.074269 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" Nov 03 20:06:21.088498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.088465 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-maas-api-xks-ca\") pod \"615f29b1-a11e-4b5d-a89b-dfefb9b91c28\" (UID: \"615f29b1-a11e-4b5d-a89b-dfefb9b91c28\") " Nov 03 20:06:21.088661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.088616 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-kube-api-access-rd2wh\" (UniqueName: \"kubernetes.io/projected/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-kube-api-access-rd2wh\") pod \"615f29b1-a11e-4b5d-a89b-dfefb9b91c28\" (UID: \"615f29b1-a11e-4b5d-a89b-dfefb9b91c28\") " Nov 03 20:06:21.088931 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.088897 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-maas-api-openshift-ca\") pod \"615f29b1-a11e-4b5d-a89b-dfefb9b91c28\" (UID: \"615f29b1-a11e-4b5d-a89b-dfefb9b91c28\") " Nov 03 20:06:21.089263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.089242 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-maas-api-openshift-ca" pod "615f29b1-a11e-4b5d-a89b-dfefb9b91c28" (UID: "615f29b1-a11e-4b5d-a89b-dfefb9b91c28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:21.089543 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.089515 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:21.090682 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.090658 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-maas-api-xks-ca" pod "615f29b1-a11e-4b5d-a89b-dfefb9b91c28" (UID: "615f29b1-a11e-4b5d-a89b-dfefb9b91c28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:21.090682 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.090666 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-kube-api-access-rd2wh" pod "615f29b1-a11e-4b5d-a89b-dfefb9b91c28" (UID: "615f29b1-a11e-4b5d-a89b-dfefb9b91c28"). InnerVolumeSpecName "kube-api-access-rd2wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:21.170416 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.170372 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq"] Nov 03 20:06:21.171560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.171515 5111 cpu_manager.go:414] "containerMap: removing container" podUID="615f29b1-a11e-4b5d-a89b-dfefb9b91c28" containerName="revoke-keys" Nov 03 20:06:21.171560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.171560 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615f29b1-a11e-4b5d-a89b-dfefb9b91c28" containerName="revoke-keys" Nov 03 20:06:21.171748 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.171586 5111 cpu_manager.go:414] "containerMap: removing container" podUID="ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e" containerName="revoke-keys" Nov 03 20:06:21.171748 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.171594 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e" containerName="revoke-keys" Nov 03 20:06:21.171885 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.171872 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="615f29b1-a11e-4b5d-a89b-dfefb9b91c28" containerName="revoke-keys" Nov 03 20:06:21.171941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.171890 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e" containerName="revoke-keys" Nov 03 20:06:21.182608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.182521 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" Nov 03 20:06:21.182807 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.182785 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq"] Nov 03 20:06:21.191199 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.191168 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hs59\" (UniqueName: \"kubernetes.io/projected/9eb7c810-e130-473f-a38a-9187120e5518-kube-api-access-2hs59\") pod \"payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq\" (UID: \"9eb7c810-e130-473f-a38a-9187120e5518\") " pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" Nov 03 20:06:21.191481 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.191457 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb7c810-e130-473f-a38a-9187120e5518-plugins-config-volume\") pod \"payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq\" (UID: \"9eb7c810-e130-473f-a38a-9187120e5518\") " pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" Nov 03 20:06:21.191658 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.191643 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd2wh\" (UniqueName: \"kubernetes.io/projected/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-kube-api-access-rd2wh\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:21.191752 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.191664 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615f29b1-a11e-4b5d-a89b-dfefb9b91c28-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:21.293610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.259497 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8fdab3c3-dockercfg-l46xr\"" Nov 03 20:06:21.293610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.259846 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8fdab3c3\"" Nov 03 20:06:21.293610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.292936 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb7c810-e130-473f-a38a-9187120e5518-plugins-config-volume\") pod \"payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq\" (UID: \"9eb7c810-e130-473f-a38a-9187120e5518\") " pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" Nov 03 20:06:21.293610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.293118 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hs59\" (UniqueName: \"kubernetes.io/projected/9eb7c810-e130-473f-a38a-9187120e5518-kube-api-access-2hs59\") pod \"payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq\" (UID: \"9eb7c810-e130-473f-a38a-9187120e5518\") " pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" Nov 03 20:06:21.302556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.295710 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb7c810-e130-473f-a38a-9187120e5518-plugins-config-volume\") pod \"payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq\" (UID: \"9eb7c810-e130-473f-a38a-9187120e5518\") " pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" Nov 03 20:06:21.326562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.326069 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hs59\" (UniqueName: \"kubernetes.io/projected/9eb7c810-e130-473f-a38a-9187120e5518-kube-api-access-2hs59\") pod \"payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq\" (UID: \"9eb7c810-e130-473f-a38a-9187120e5518\") " pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" Nov 03 20:06:21.519607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.509903 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" Nov 03 20:06:21.876484 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.876442 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" event={"ID":"615f29b1-a11e-4b5d-a89b-dfefb9b91c28","Type":"ContainerDied","Data":"93774d808f02d27824a3a1764f5b091f98f6b516ee703db59a57037188b83f02"} Nov 03 20:06:21.877028 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.876557 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93774d808f02d27824a3a1764f5b091f98f6b516ee703db59a57037188b83f02" Nov 03 20:06:21.877028 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.876684 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw" Nov 03 20:06:21.903883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.903857 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:21.910268 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:06:21.910240 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb7c810_e130_473f_a38a_9187120e5518.slice/crio-38f46963912ac8cfdbe4516222ecfc79f8833608a8159e17bb3b89493683cd29 WatchSource:0}: Error finding container 38f46963912ac8cfdbe4516222ecfc79f8833608a8159e17bb3b89493683cd29: Status 404 returned error can't find the container with id 38f46963912ac8cfdbe4516222ecfc79f8833608a8159e17bb3b89493683cd29 Nov 03 20:06:21.914889 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:21.914866 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq"] Nov 03 20:06:22.889526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:22.889491 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" event={"ID":"9eb7c810-e130-473f-a38a-9187120e5518","Type":"ContainerStarted","Data":"1ac6142633ef82e20b0c7627af57e21cffefbcc38962a1ac093e74ca8ae962d8"} Nov 03 20:06:22.890016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:22.889564 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" event={"ID":"9eb7c810-e130-473f-a38a-9187120e5518","Type":"ContainerStarted","Data":"38f46963912ac8cfdbe4516222ecfc79f8833608a8159e17bb3b89493683cd29"} Nov 03 20:06:22.890016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:22.889593 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" Nov 03 20:06:22.890131 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:22.890061 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:22.890963 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:22.890940 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:06:22.942594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:22.942490 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" podStartSLOduration=1.942475121 podStartE2EDuration="1.942475121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:22.937727788 +0000 UTC m=+1612.900106219" watchObservedRunningTime="2026-11-03 20:06:22.942475121 +0000 UTC m=+1612.904853655" Nov 03 20:06:23.807657 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:23.807625 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f"] Nov 03 20:06:23.812432 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:23.812413 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f" Nov 03 20:06:23.831171 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:23.831146 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:06:23.831676 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:23.831525 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:06:23.831676 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:23.831650 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:06:23.831847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:23.831818 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:06:23.844430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:23.844403 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f"] Nov 03 20:06:23.920217 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:23.920184 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7e2d25-c018-4169-9dca-165fc108463a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fd09552-5mh2f\" (UID: \"0a7e2d25-c018-4169-9dca-165fc108463a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f" Nov 03 20:06:23.920601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:23.920316 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gdth\" (UniqueName: \"kubernetes.io/projected/0a7e2d25-c018-4169-9dca-165fc108463a-kube-api-access-7gdth\") pod \"maas-api-revoke-sub-5fd09552-5mh2f\" (UID: \"0a7e2d25-c018-4169-9dca-165fc108463a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f" Nov 03 20:06:23.920601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:23.920466 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7e2d25-c018-4169-9dca-165fc108463a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fd09552-5mh2f\" (UID: \"0a7e2d25-c018-4169-9dca-165fc108463a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f" Nov 03 20:06:24.021824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.021786 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7e2d25-c018-4169-9dca-165fc108463a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fd09552-5mh2f\" (UID: \"0a7e2d25-c018-4169-9dca-165fc108463a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f" Nov 03 20:06:24.021996 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.021977 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7e2d25-c018-4169-9dca-165fc108463a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fd09552-5mh2f\" (UID: \"0a7e2d25-c018-4169-9dca-165fc108463a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f" Nov 03 20:06:24.022143 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.022123 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gdth\" (UniqueName: \"kubernetes.io/projected/0a7e2d25-c018-4169-9dca-165fc108463a-kube-api-access-7gdth\") pod \"maas-api-revoke-sub-5fd09552-5mh2f\" (UID: \"0a7e2d25-c018-4169-9dca-165fc108463a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f" Nov 03 20:06:24.022631 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.022601 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7e2d25-c018-4169-9dca-165fc108463a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fd09552-5mh2f\" (UID: \"0a7e2d25-c018-4169-9dca-165fc108463a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f" Nov 03 20:06:24.024365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.024340 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7e2d25-c018-4169-9dca-165fc108463a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fd09552-5mh2f\" (UID: \"0a7e2d25-c018-4169-9dca-165fc108463a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f" Nov 03 20:06:24.046743 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.046713 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gdth\" (UniqueName: \"kubernetes.io/projected/0a7e2d25-c018-4169-9dca-165fc108463a-kube-api-access-7gdth\") pod \"maas-api-revoke-sub-5fd09552-5mh2f\" (UID: \"0a7e2d25-c018-4169-9dca-165fc108463a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f" Nov 03 20:06:24.128331 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.128298 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f" Nov 03 20:06:24.319033 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.318987 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv"] Nov 03 20:06:24.325242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.325217 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" Nov 03 20:06:24.333073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.333038 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ea700cd3\"" Nov 03 20:06:24.334270 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.334249 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ea700cd3-dockercfg-nvsjv\"" Nov 03 20:06:24.341868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.341691 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv"] Nov 03 20:06:24.383053 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.382968 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt"] Nov 03 20:06:24.390277 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.390250 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" Nov 03 20:06:24.390488 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.390462 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt"] Nov 03 20:06:24.731635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.731604 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f"] Nov 03 20:06:24.732482 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.626850 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b0cbea-8030-4b84-aa0f-c886f8f7b550-plugins-config-volume\") pod \"payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv\" (UID: \"89b0cbea-8030-4b84-aa0f-c886f8f7b550\") " pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" Nov 03 20:06:24.732623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.732592 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbljb\" (UniqueName: \"kubernetes.io/projected/89b0cbea-8030-4b84-aa0f-c886f8f7b550-kube-api-access-rbljb\") pod \"payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv\" (UID: \"89b0cbea-8030-4b84-aa0f-c886f8f7b550\") " pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" Nov 03 20:06:24.833743 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.833705 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20828b4f-f0b4-4ab4-95ed-646f052a2e76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt\" (UID: \"20828b4f-f0b4-4ab4-95ed-646f052a2e76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" Nov 03 20:06:24.833946 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.833922 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftqrw\" (UniqueName: \"kubernetes.io/projected/20828b4f-f0b4-4ab4-95ed-646f052a2e76-kube-api-access-ftqrw\") pod \"payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt\" (UID: \"20828b4f-f0b4-4ab4-95ed-646f052a2e76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" Nov 03 20:06:24.834038 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.834021 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b0cbea-8030-4b84-aa0f-c886f8f7b550-plugins-config-volume\") pod \"payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv\" (UID: \"89b0cbea-8030-4b84-aa0f-c886f8f7b550\") " pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" Nov 03 20:06:24.834100 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.834069 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbljb\" (UniqueName: \"kubernetes.io/projected/89b0cbea-8030-4b84-aa0f-c886f8f7b550-kube-api-access-rbljb\") pod \"payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv\" (UID: \"89b0cbea-8030-4b84-aa0f-c886f8f7b550\") " pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" Nov 03 20:06:24.835101 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.835073 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b0cbea-8030-4b84-aa0f-c886f8f7b550-plugins-config-volume\") pod \"payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv\" (UID: \"89b0cbea-8030-4b84-aa0f-c886f8f7b550\") " pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" Nov 03 20:06:24.843991 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.843965 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbljb\" (UniqueName: \"kubernetes.io/projected/89b0cbea-8030-4b84-aa0f-c886f8f7b550-kube-api-access-rbljb\") pod \"payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv\" (UID: \"89b0cbea-8030-4b84-aa0f-c886f8f7b550\") " pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" Nov 03 20:06:24.913748 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.913658 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f" event={"ID":"0a7e2d25-c018-4169-9dca-165fc108463a","Type":"ContainerStarted","Data":"e4f86b0b01d86f4ed5a3b73d0c0cd9ded2940c3281aba3ecd661509e6ec68024"} Nov 03 20:06:24.939894 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.939845 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20828b4f-f0b4-4ab4-95ed-646f052a2e76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt\" (UID: \"20828b4f-f0b4-4ab4-95ed-646f052a2e76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" Nov 03 20:06:24.940365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.940044 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftqrw\" (UniqueName: \"kubernetes.io/projected/20828b4f-f0b4-4ab4-95ed-646f052a2e76-kube-api-access-ftqrw\") pod \"payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt\" (UID: \"20828b4f-f0b4-4ab4-95ed-646f052a2e76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" Nov 03 20:06:24.940901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.940870 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20828b4f-f0b4-4ab4-95ed-646f052a2e76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt\" (UID: \"20828b4f-f0b4-4ab4-95ed-646f052a2e76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" Nov 03 20:06:24.944920 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.944896 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" Nov 03 20:06:24.950008 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:24.949986 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftqrw\" (UniqueName: \"kubernetes.io/projected/20828b4f-f0b4-4ab4-95ed-646f052a2e76-kube-api-access-ftqrw\") pod \"payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt\" (UID: \"20828b4f-f0b4-4ab4-95ed-646f052a2e76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" Nov 03 20:06:25.009824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:25.009786 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" Nov 03 20:06:25.284955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:25.284910 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv"] Nov 03 20:06:25.285463 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:06:25.285421 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b0cbea_8030_4b84_aa0f_c886f8f7b550.slice/crio-3d52cbf5ded8da986ca9fbccac1c48697f69d6800537731377b58f629191bcde WatchSource:0}: Error finding container 3d52cbf5ded8da986ca9fbccac1c48697f69d6800537731377b58f629191bcde: Status 404 returned error can't find the container with id 3d52cbf5ded8da986ca9fbccac1c48697f69d6800537731377b58f629191bcde Nov 03 20:06:25.460155 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:06:25.460123 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20828b4f_f0b4_4ab4_95ed_646f052a2e76.slice/crio-1838c9a3d7d6a34027fbd1ff95d3e8ebadbc0bcb0ae1cd2a491795bd6c1c049e WatchSource:0}: Error finding container 1838c9a3d7d6a34027fbd1ff95d3e8ebadbc0bcb0ae1cd2a491795bd6c1c049e: Status 404 returned error can't find the container with id 1838c9a3d7d6a34027fbd1ff95d3e8ebadbc0bcb0ae1cd2a491795bd6c1c049e Nov 03 20:06:25.463665 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:25.463635 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt"] Nov 03 20:06:25.979557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:25.966026 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" event={"ID":"89b0cbea-8030-4b84-aa0f-c886f8f7b550","Type":"ContainerStarted","Data":"c72896e26419e1307a1fe3055f370c7f9f427224cb7138f9d2a03b52e78d400e"} Nov 03 20:06:25.979557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:25.966125 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" event={"ID":"89b0cbea-8030-4b84-aa0f-c886f8f7b550","Type":"ContainerStarted","Data":"3d52cbf5ded8da986ca9fbccac1c48697f69d6800537731377b58f629191bcde"} Nov 03 20:06:25.979557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:25.967362 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" Nov 03 20:06:26.021612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.012174 5111 generic.go:356] "Generic (PLEG): container finished" podID="0a7e2d25-c018-4169-9dca-165fc108463a" containerID="f1f83b03e29a1eb932c2cef92195f904374c5dd37e7f562a110e5969098babef" exitCode=0 Nov 03 20:06:26.021612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.012400 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f" event={"ID":"0a7e2d25-c018-4169-9dca-165fc108463a","Type":"ContainerDied","Data":"f1f83b03e29a1eb932c2cef92195f904374c5dd37e7f562a110e5969098babef"} Nov 03 20:06:26.021612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.016470 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" event={"ID":"20828b4f-f0b4-4ab4-95ed-646f052a2e76","Type":"ContainerStarted","Data":"8a562eff23e5e6911627291ccde567aad0e124f80d6d1619e5aad2dff08518c4"} Nov 03 20:06:26.021612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.016525 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" event={"ID":"20828b4f-f0b4-4ab4-95ed-646f052a2e76","Type":"ContainerStarted","Data":"1838c9a3d7d6a34027fbd1ff95d3e8ebadbc0bcb0ae1cd2a491795bd6c1c049e"} Nov 03 20:06:26.021612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.018128 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" Nov 03 20:06:26.021612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.020629 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" podStartSLOduration=2.020616072 podStartE2EDuration="2.020616072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:26.017748475 +0000 UTC m=+1615.980126916" watchObservedRunningTime="2026-11-03 20:06:26.020616072 +0000 UTC m=+1615.982994515" Nov 03 20:06:26.110707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.110640 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" podStartSLOduration=2.110619033 podStartE2EDuration="2.110619033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:26.1058703 +0000 UTC m=+1616.068248740" watchObservedRunningTime="2026-11-03 20:06:26.110619033 +0000 UTC m=+1616.072997473" Nov 03 20:06:26.453933 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.453898 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4"] Nov 03 20:06:26.460441 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.460115 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" Nov 03 20:06:26.487627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.487589 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4"] Nov 03 20:06:26.524767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.524728 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8440d1f-afd2-4f2d-b17e-cbb699b51444-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65543e7a-rp6c4\" (UID: \"b8440d1f-afd2-4f2d-b17e-cbb699b51444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" Nov 03 20:06:26.524962 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.524940 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b658w\" (UniqueName: \"kubernetes.io/projected/b8440d1f-afd2-4f2d-b17e-cbb699b51444-kube-api-access-b658w\") pod \"maas-api-revoke-sub-65543e7a-rp6c4\" (UID: \"b8440d1f-afd2-4f2d-b17e-cbb699b51444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" Nov 03 20:06:26.525263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.525237 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8440d1f-afd2-4f2d-b17e-cbb699b51444-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65543e7a-rp6c4\" (UID: \"b8440d1f-afd2-4f2d-b17e-cbb699b51444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" Nov 03 20:06:26.627224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.627188 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b658w\" (UniqueName: \"kubernetes.io/projected/b8440d1f-afd2-4f2d-b17e-cbb699b51444-kube-api-access-b658w\") pod \"maas-api-revoke-sub-65543e7a-rp6c4\" (UID: \"b8440d1f-afd2-4f2d-b17e-cbb699b51444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" Nov 03 20:06:26.627411 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.627376 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8440d1f-afd2-4f2d-b17e-cbb699b51444-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65543e7a-rp6c4\" (UID: \"b8440d1f-afd2-4f2d-b17e-cbb699b51444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" Nov 03 20:06:26.627475 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.627414 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8440d1f-afd2-4f2d-b17e-cbb699b51444-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65543e7a-rp6c4\" (UID: \"b8440d1f-afd2-4f2d-b17e-cbb699b51444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" Nov 03 20:06:26.628171 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.628144 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8440d1f-afd2-4f2d-b17e-cbb699b51444-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65543e7a-rp6c4\" (UID: \"b8440d1f-afd2-4f2d-b17e-cbb699b51444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" Nov 03 20:06:26.629823 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.629800 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8440d1f-afd2-4f2d-b17e-cbb699b51444-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65543e7a-rp6c4\" (UID: \"b8440d1f-afd2-4f2d-b17e-cbb699b51444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" Nov 03 20:06:26.636937 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.636912 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b658w\" (UniqueName: \"kubernetes.io/projected/b8440d1f-afd2-4f2d-b17e-cbb699b51444-kube-api-access-b658w\") pod \"maas-api-revoke-sub-65543e7a-rp6c4\" (UID: \"b8440d1f-afd2-4f2d-b17e-cbb699b51444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" Nov 03 20:06:26.811352 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:26.811273 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" Nov 03 20:06:27.124834 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:06:27.124784 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8440d1f_afd2_4f2d_b17e_cbb699b51444.slice/crio-6fdb322eaa0746b3dd7bb90fbc8e06c3c64c00378673f74a27fa6d9457e8b194 WatchSource:0}: Error finding container 6fdb322eaa0746b3dd7bb90fbc8e06c3c64c00378673f74a27fa6d9457e8b194: Status 404 returned error can't find the container with id 6fdb322eaa0746b3dd7bb90fbc8e06c3c64c00378673f74a27fa6d9457e8b194 Nov 03 20:06:27.190513 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.189994 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4"] Nov 03 20:06:27.283384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.283343 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f" Nov 03 20:06:27.375520 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.374061 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7e2d25-c018-4169-9dca-165fc108463a-kube-api-access-7gdth\" (UniqueName: \"kubernetes.io/projected/0a7e2d25-c018-4169-9dca-165fc108463a-kube-api-access-7gdth\") pod \"0a7e2d25-c018-4169-9dca-165fc108463a\" (UID: \"0a7e2d25-c018-4169-9dca-165fc108463a\") " Nov 03 20:06:27.375520 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.374157 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a7e2d25-c018-4169-9dca-165fc108463a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7e2d25-c018-4169-9dca-165fc108463a-maas-api-xks-ca\") pod \"0a7e2d25-c018-4169-9dca-165fc108463a\" (UID: \"0a7e2d25-c018-4169-9dca-165fc108463a\") " Nov 03 20:06:27.375520 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.374612 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a7e2d25-c018-4169-9dca-165fc108463a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7e2d25-c018-4169-9dca-165fc108463a-maas-api-openshift-ca\") pod \"0a7e2d25-c018-4169-9dca-165fc108463a\" (UID: \"0a7e2d25-c018-4169-9dca-165fc108463a\") " Nov 03 20:06:27.375520 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.375185 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7e2d25-c018-4169-9dca-165fc108463a-maas-api-openshift-ca" pod "0a7e2d25-c018-4169-9dca-165fc108463a" (UID: "0a7e2d25-c018-4169-9dca-165fc108463a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:27.375849 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.375572 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7e2d25-c018-4169-9dca-165fc108463a-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:27.377232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.377202 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a7e2d25-c018-4169-9dca-165fc108463a-maas-api-xks-ca" pod "0a7e2d25-c018-4169-9dca-165fc108463a" (UID: "0a7e2d25-c018-4169-9dca-165fc108463a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:27.378065 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.378035 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7e2d25-c018-4169-9dca-165fc108463a-kube-api-access-7gdth" pod "0a7e2d25-c018-4169-9dca-165fc108463a" (UID: "0a7e2d25-c018-4169-9dca-165fc108463a"). InnerVolumeSpecName "kube-api-access-7gdth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:27.477570 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.477506 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gdth\" (UniqueName: \"kubernetes.io/projected/0a7e2d25-c018-4169-9dca-165fc108463a-kube-api-access-7gdth\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:27.477570 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.477572 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7e2d25-c018-4169-9dca-165fc108463a-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:27.728217 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.728183 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c"] Nov 03 20:06:27.729335 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.729308 5111 cpu_manager.go:414] "containerMap: removing container" podUID="0a7e2d25-c018-4169-9dca-165fc108463a" containerName="revoke-keys" Nov 03 20:06:27.729335 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.729330 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7e2d25-c018-4169-9dca-165fc108463a" containerName="revoke-keys" Nov 03 20:06:27.729590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.729506 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7e2d25-c018-4169-9dca-165fc108463a" containerName="revoke-keys" Nov 03 20:06:27.734619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.734596 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" Nov 03 20:06:27.739748 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.739723 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c"] Nov 03 20:06:27.884637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.884597 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51f9b761-7l27c\" (UID: \"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" Nov 03 20:06:27.884836 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.884809 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51f9b761-7l27c\" (UID: \"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" Nov 03 20:06:27.885006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.884988 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mks7g\" (UniqueName: \"kubernetes.io/projected/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-kube-api-access-mks7g\") pod \"maas-api-revoke-sub-51f9b761-7l27c\" (UID: \"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" Nov 03 20:06:27.987095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.987007 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51f9b761-7l27c\" (UID: \"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" Nov 03 20:06:27.987275 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.987172 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51f9b761-7l27c\" (UID: \"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" Nov 03 20:06:27.987333 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.987317 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mks7g\" (UniqueName: \"kubernetes.io/projected/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-kube-api-access-mks7g\") pod \"maas-api-revoke-sub-51f9b761-7l27c\" (UID: \"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" Nov 03 20:06:27.987923 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.987896 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51f9b761-7l27c\" (UID: \"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" Nov 03 20:06:27.989623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.989602 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51f9b761-7l27c\" (UID: \"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" Nov 03 20:06:27.996301 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:27.996274 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mks7g\" (UniqueName: \"kubernetes.io/projected/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-kube-api-access-mks7g\") pod \"maas-api-revoke-sub-51f9b761-7l27c\" (UID: \"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" Nov 03 20:06:28.047526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:28.047500 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f" Nov 03 20:06:28.047713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:28.047520 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f" event={"ID":"0a7e2d25-c018-4169-9dca-165fc108463a","Type":"ContainerDied","Data":"e4f86b0b01d86f4ed5a3b73d0c0cd9ded2940c3281aba3ecd661509e6ec68024"} Nov 03 20:06:28.047713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:28.047594 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4f86b0b01d86f4ed5a3b73d0c0cd9ded2940c3281aba3ecd661509e6ec68024" Nov 03 20:06:28.049773 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:28.049745 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" event={"ID":"b8440d1f-afd2-4f2d-b17e-cbb699b51444","Type":"ContainerStarted","Data":"1b6ad5e3d968fadf93077cca3da8d0fc44b3f01f254739dde3c5e48d2d0fdb1e"} Nov 03 20:06:28.049908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:28.049794 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" event={"ID":"b8440d1f-afd2-4f2d-b17e-cbb699b51444","Type":"ContainerStarted","Data":"6fdb322eaa0746b3dd7bb90fbc8e06c3c64c00378673f74a27fa6d9457e8b194"} Nov 03 20:06:28.055671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:28.055650 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" Nov 03 20:06:28.074127 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:28.074086 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" podStartSLOduration=2.074074973 podStartE2EDuration="2.074074973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:28.067721566 +0000 UTC m=+1618.030100008" watchObservedRunningTime="2026-11-03 20:06:28.074074973 +0000 UTC m=+1618.036453414" Nov 03 20:06:28.369643 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:06:28.369608 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22218915_cc64_4ed1_8cc9_9f4aef1d9c5a.slice/crio-16da957b93d0d6262b0f4de64e99b48637441970a13309c2b7187b651d838480 WatchSource:0}: Error finding container 16da957b93d0d6262b0f4de64e99b48637441970a13309c2b7187b651d838480: Status 404 returned error can't find the container with id 16da957b93d0d6262b0f4de64e99b48637441970a13309c2b7187b651d838480 Nov 03 20:06:28.384110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:28.384082 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c"] Nov 03 20:06:29.061407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:29.061367 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" event={"ID":"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a","Type":"ContainerStarted","Data":"d22832cf73518f31f9ceedfd55f315590a93d6301fcdc08dab2a5147ed5aab52"} Nov 03 20:06:29.061631 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:29.061435 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" event={"ID":"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a","Type":"ContainerStarted","Data":"16da957b93d0d6262b0f4de64e99b48637441970a13309c2b7187b651d838480"} Nov 03 20:06:29.063324 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:29.063293 5111 generic.go:356] "Generic (PLEG): container finished" podID="b8440d1f-afd2-4f2d-b17e-cbb699b51444" containerID="1b6ad5e3d968fadf93077cca3da8d0fc44b3f01f254739dde3c5e48d2d0fdb1e" exitCode=0 Nov 03 20:06:29.063463 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:29.063379 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" event={"ID":"b8440d1f-afd2-4f2d-b17e-cbb699b51444","Type":"ContainerDied","Data":"1b6ad5e3d968fadf93077cca3da8d0fc44b3f01f254739dde3c5e48d2d0fdb1e"} Nov 03 20:06:29.090291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:29.090225 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" podStartSLOduration=2.0902109700000002 podStartE2EDuration="2.09021097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:29.082222228 +0000 UTC m=+1619.044600683" watchObservedRunningTime="2026-11-03 20:06:29.09021097 +0000 UTC m=+1619.052589425" Nov 03 20:06:30.076385 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:30.076353 5111 generic.go:356] "Generic (PLEG): container finished" podID="22218915-cc64-4ed1-8cc9-9f4aef1d9c5a" containerID="d22832cf73518f31f9ceedfd55f315590a93d6301fcdc08dab2a5147ed5aab52" exitCode=0 Nov 03 20:06:30.076937 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:30.076445 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" event={"ID":"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a","Type":"ContainerDied","Data":"d22832cf73518f31f9ceedfd55f315590a93d6301fcdc08dab2a5147ed5aab52"} Nov 03 20:06:30.510220 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:30.508713 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" Nov 03 20:06:30.673649 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:30.673616 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8440d1f-afd2-4f2d-b17e-cbb699b51444-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8440d1f-afd2-4f2d-b17e-cbb699b51444-maas-api-xks-ca\") pod \"b8440d1f-afd2-4f2d-b17e-cbb699b51444\" (UID: \"b8440d1f-afd2-4f2d-b17e-cbb699b51444\") " Nov 03 20:06:30.673842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:30.673657 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8440d1f-afd2-4f2d-b17e-cbb699b51444-kube-api-access-b658w\" (UniqueName: \"kubernetes.io/projected/b8440d1f-afd2-4f2d-b17e-cbb699b51444-kube-api-access-b658w\") pod \"b8440d1f-afd2-4f2d-b17e-cbb699b51444\" (UID: \"b8440d1f-afd2-4f2d-b17e-cbb699b51444\") " Nov 03 20:06:30.673842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:30.673711 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8440d1f-afd2-4f2d-b17e-cbb699b51444-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8440d1f-afd2-4f2d-b17e-cbb699b51444-maas-api-openshift-ca\") pod \"b8440d1f-afd2-4f2d-b17e-cbb699b51444\" (UID: \"b8440d1f-afd2-4f2d-b17e-cbb699b51444\") " Nov 03 20:06:30.674153 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:30.674127 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8440d1f-afd2-4f2d-b17e-cbb699b51444-maas-api-openshift-ca" pod "b8440d1f-afd2-4f2d-b17e-cbb699b51444" (UID: "b8440d1f-afd2-4f2d-b17e-cbb699b51444"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:30.674517 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:30.674498 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8440d1f-afd2-4f2d-b17e-cbb699b51444-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:30.675929 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:30.675901 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8440d1f-afd2-4f2d-b17e-cbb699b51444-kube-api-access-b658w" pod "b8440d1f-afd2-4f2d-b17e-cbb699b51444" (UID: "b8440d1f-afd2-4f2d-b17e-cbb699b51444"). InnerVolumeSpecName "kube-api-access-b658w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:30.675929 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:30.675918 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8440d1f-afd2-4f2d-b17e-cbb699b51444-maas-api-xks-ca" pod "b8440d1f-afd2-4f2d-b17e-cbb699b51444" (UID: "b8440d1f-afd2-4f2d-b17e-cbb699b51444"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:30.775732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:30.775695 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8440d1f-afd2-4f2d-b17e-cbb699b51444-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:30.775732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:30.775727 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b658w\" (UniqueName: \"kubernetes.io/projected/b8440d1f-afd2-4f2d-b17e-cbb699b51444-kube-api-access-b658w\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:31.089028 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:31.088951 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" Nov 03 20:06:31.089028 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:31.088992 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4" event={"ID":"b8440d1f-afd2-4f2d-b17e-cbb699b51444","Type":"ContainerDied","Data":"6fdb322eaa0746b3dd7bb90fbc8e06c3c64c00378673f74a27fa6d9457e8b194"} Nov 03 20:06:31.089605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:31.089050 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fdb322eaa0746b3dd7bb90fbc8e06c3c64c00378673f74a27fa6d9457e8b194" Nov 03 20:06:31.273499 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:31.273475 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" Nov 03 20:06:31.293235 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:31.293184 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-maas-api-xks-ca\") pod \"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a\" (UID: \"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a\") " Nov 03 20:06:31.293413 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:31.293328 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-kube-api-access-mks7g\" (UniqueName: \"kubernetes.io/projected/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-kube-api-access-mks7g\") pod \"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a\" (UID: \"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a\") " Nov 03 20:06:31.293413 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:31.293382 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-maas-api-openshift-ca\") pod \"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a\" (UID: \"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a\") " Nov 03 20:06:31.303417 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:31.303031 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-maas-api-openshift-ca" pod "22218915-cc64-4ed1-8cc9-9f4aef1d9c5a" (UID: "22218915-cc64-4ed1-8cc9-9f4aef1d9c5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:31.336205 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:31.336076 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-maas-api-xks-ca" pod "22218915-cc64-4ed1-8cc9-9f4aef1d9c5a" (UID: "22218915-cc64-4ed1-8cc9-9f4aef1d9c5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:31.336205 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:31.336138 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-kube-api-access-mks7g" pod "22218915-cc64-4ed1-8cc9-9f4aef1d9c5a" (UID: "22218915-cc64-4ed1-8cc9-9f4aef1d9c5a"). InnerVolumeSpecName "kube-api-access-mks7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:31.396557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:31.396506 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:31.396557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:31.396553 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mks7g\" (UniqueName: \"kubernetes.io/projected/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-kube-api-access-mks7g\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:31.396855 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:31.396574 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:32.099620 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:32.099587 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" Nov 03 20:06:32.100209 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:32.099623 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c" event={"ID":"22218915-cc64-4ed1-8cc9-9f4aef1d9c5a","Type":"ContainerDied","Data":"16da957b93d0d6262b0f4de64e99b48637441970a13309c2b7187b651d838480"} Nov 03 20:06:32.100209 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:32.099681 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16da957b93d0d6262b0f4de64e99b48637441970a13309c2b7187b651d838480" Nov 03 20:06:33.474699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:33.450630 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:06:33.495076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:33.493816 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:33.639560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:33.636035 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:33.731101 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:33.730993 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:33.793566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:33.739307 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:33.842734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:33.842675 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:34.045614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:33.970752 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" Nov 03 20:06:34.116442 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.115636 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:34.171573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.154788 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:34.352650 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.352063 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:34.651648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.489863 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:34.651648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.520342 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:34.651648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.520725 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:06:34.651648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.544706 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:06:34.652347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.651841 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:06:34.688013 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.686687 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:34.688013 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.687269 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 03 20:06:34.771554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.771076 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:34.795569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.781221 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:34.837560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.835160 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:06:34.837560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.835237 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:06:34.837560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.835278 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:06:34.837560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.835316 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:06:34.837560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.835351 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:06:34.858725 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.858669 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:34.878124 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:34.878080 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:06:35.206071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.135631 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:35.206071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.136075 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:06:35.206071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.155994 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:35.206071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.169773 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:06:35.354554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.353024 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:06:35.378703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.378662 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:06:35.408558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.406671 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:35.467614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.457668 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded" Nov 03 20:06:35.532494 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.511481 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:35.532494 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.521861 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:06:35.595006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.561356 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:35.595006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.564664 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:06:35.595006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.565204 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:06:35.621352 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.596717 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:35.808489 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.782755 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:35.808489 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.797691 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:35.872525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.839499 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:35.894593 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:35.891127 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:36.147678 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:36.147625 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:36.206558 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:06:36.157555 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:06:36.292060 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:36.166723 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:36.494620 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:36.441642 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:06:36.586770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:36.586718 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 20:06:36.588071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:36.588038 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:36.589139 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:36.589116 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:06:36.609608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:36.609563 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:36.821837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:36.802376 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:36.826903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:36.826864 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:06:36.855397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:36.853863 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:06:36.856221 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:36.855687 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Nov 03 20:06:36.916559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:36.909838 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:37.373056 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:37.372995 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:37.384078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:37.384031 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 20:06:37.589615 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:37.564511 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 03 20:06:37.589860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:37.573634 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:06:37.669616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:37.662926 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded" Nov 03 20:06:37.860559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:37.854920 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:37.927626 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:37.910360 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:37.927626 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:37.910838 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:38.108562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:38.107614 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:38.184275 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:38.184144 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 20:06:38.209833 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:38.209785 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:06:38.485720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:38.470296 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" podUID="1e292de6-b36f-481b-9d69-1961e7d8026b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:38.486005 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:38.478936 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:06:38.598223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:38.598175 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 20:06:38.617859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:38.609945 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:06:38.709971 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:38.699665 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:38.801655 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:38.772770 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" podUID="89b0cbea-8030-4b84-aa0f-c886f8f7b550" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.150:50052\" within 1s: context deadline exceeded" Nov 03 20:06:38.801655 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:06:38.772991 5111 logging.go:55] [core] [Channel #817 SubChannel #821]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.150:50052", ServerName: "10.132.0.150:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.150:50052: operation was canceled" Nov 03 20:06:38.801655 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:06:38.775980 5111 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.526s" Nov 03 20:06:38.832753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:38.832686 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:38.864828 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:38.864783 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:38.913592 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:38.911463 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:38.960135 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:38.926423 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 20:06:39.106678 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.101906 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" Nov 03 20:06:39.134559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.128757 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:06:39.198701 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.198648 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:39.221556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.215885 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:39.357770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.357674 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:06:39.395559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.368611 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" podUID="87cff690-9d53-44d6-a1dc-3f96238ca235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:39.617243 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.617143 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded" Nov 03 20:06:39.622926 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.622622 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 20:06:39.688158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.622671 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:39.712981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.712937 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:39.806670 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.806623 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:39.866930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.866818 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:39.900525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.900481 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:06:39.901891 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.901181 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:06:39.914284 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.914255 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:06:39.975119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.975068 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:39.976172 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:39.975500 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:40.141679 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:40.141619 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:06:40.167559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:40.167384 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:06:40.216582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:40.213743 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:40.223484 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:40.222572 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:06:40.322840 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:40.322809 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:06:40.323075 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:40.323061 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:06:40.323178 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:40.323169 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:06:40.340638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:40.339171 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:06:40.349631 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:40.341777 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:06:40.362332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:40.362307 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:06:40.365739 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:40.365716 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:06:40.365996 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:40.365982 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:06:47.039246 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:47.039215 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" Nov 03 20:06:50.730849 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.730815 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg"] Nov 03 20:06:50.732210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.731611 5111 cpu_manager.go:414] "containerMap: removing container" podUID="b8440d1f-afd2-4f2d-b17e-cbb699b51444" containerName="revoke-keys" Nov 03 20:06:50.732210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.731626 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8440d1f-afd2-4f2d-b17e-cbb699b51444" containerName="revoke-keys" Nov 03 20:06:50.732210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.731666 5111 cpu_manager.go:414] "containerMap: removing container" podUID="22218915-cc64-4ed1-8cc9-9f4aef1d9c5a" containerName="revoke-keys" Nov 03 20:06:50.732210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.731671 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22218915-cc64-4ed1-8cc9-9f4aef1d9c5a" containerName="revoke-keys" Nov 03 20:06:50.732210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.731779 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8440d1f-afd2-4f2d-b17e-cbb699b51444" containerName="revoke-keys" Nov 03 20:06:50.732210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.731793 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="22218915-cc64-4ed1-8cc9-9f4aef1d9c5a" containerName="revoke-keys" Nov 03 20:06:50.737398 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.737370 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" Nov 03 20:06:50.745585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.743724 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:06:50.745585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.743748 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:06:50.745585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.743778 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:06:50.745585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.743802 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:06:50.745585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.744512 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg"] Nov 03 20:06:50.878597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.878541 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg\" (UID: \"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" Nov 03 20:06:50.878978 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.878951 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-245bx\" (UniqueName: \"kubernetes.io/projected/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-kube-api-access-245bx\") pod \"maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg\" (UID: \"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" Nov 03 20:06:50.879171 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.879146 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg\" (UID: \"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" Nov 03 20:06:50.981369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.981268 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg\" (UID: \"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" Nov 03 20:06:50.981369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.981349 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg\" (UID: \"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" Nov 03 20:06:50.981626 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.981580 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-245bx\" (UniqueName: \"kubernetes.io/projected/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-kube-api-access-245bx\") pod \"maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg\" (UID: \"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" Nov 03 20:06:50.982511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.982482 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg\" (UID: \"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" Nov 03 20:06:50.988368 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.988331 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg\" (UID: \"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" Nov 03 20:06:50.990115 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:50.990063 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-245bx\" (UniqueName: \"kubernetes.io/projected/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-kube-api-access-245bx\") pod \"maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg\" (UID: \"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" Nov 03 20:06:51.065117 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:51.064809 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" Nov 03 20:06:51.383862 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:06:51.383814 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a7ed3b_fbe3_423f_aadc_1a32f8ccad73.slice/crio-40fa7546a063852edc61f4bf524491c13980660534ea21407e093e3528f45fd0 WatchSource:0}: Error finding container 40fa7546a063852edc61f4bf524491c13980660534ea21407e093e3528f45fd0: Status 404 returned error can't find the container with id 40fa7546a063852edc61f4bf524491c13980660534ea21407e093e3528f45fd0 Nov 03 20:06:51.384782 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:51.384758 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg"] Nov 03 20:06:52.058099 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.058000 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" event={"ID":"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73","Type":"ContainerStarted","Data":"bd17bfa3cf7b0297a876848ab264192a85a8bffbed32547a316895a2ba46a14a"} Nov 03 20:06:52.058099 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.058060 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" event={"ID":"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73","Type":"ContainerStarted","Data":"40fa7546a063852edc61f4bf524491c13980660534ea21407e093e3528f45fd0"} Nov 03 20:06:52.099150 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.099086 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" podStartSLOduration=2.099067504 podStartE2EDuration="2.099067504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:52.087719194 +0000 UTC m=+1642.050097632" watchObservedRunningTime="2026-11-03 20:06:52.099067504 +0000 UTC m=+1642.061445946" Nov 03 20:06:52.460604 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.460573 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7"] Nov 03 20:06:52.465588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.465569 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.476525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.476500 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7"] Nov 03 20:06:52.590681 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.590648 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g"] Nov 03 20:06:52.595656 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.595633 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" Nov 03 20:06:52.616703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.607027 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g"] Nov 03 20:06:52.616703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.608113 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.616703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.608292 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.616703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.608355 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0da8389-92b0-42da-96cd-4dec2757e02a-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.616703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.608457 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.616703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.608652 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.616703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.608683 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.616703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.608718 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.616703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.608773 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.616703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.608835 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpl6s\" (UniqueName: \"kubernetes.io/projected/a0da8389-92b0-42da-96cd-4dec2757e02a-kube-api-access-tpl6s\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.710601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.710561 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5hxq\" (UniqueName: \"kubernetes.io/projected/aed464a2-06ae-4d29-9880-9bbf2773ac1d-kube-api-access-n5hxq\") pod \"maas-api-revoke-sub-2418922c-mct8g\" (UID: \"aed464a2-06ae-4d29-9880-9bbf2773ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" Nov 03 20:06:52.710842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.710704 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed464a2-06ae-4d29-9880-9bbf2773ac1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2418922c-mct8g\" (UID: \"aed464a2-06ae-4d29-9880-9bbf2773ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" Nov 03 20:06:52.710842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.710758 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.710842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.710805 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0da8389-92b0-42da-96cd-4dec2757e02a-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.711039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.710853 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.711039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.710970 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed464a2-06ae-4d29-9880-9bbf2773ac1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2418922c-mct8g\" (UID: \"aed464a2-06ae-4d29-9880-9bbf2773ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" Nov 03 20:06:52.711039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.710994 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.711039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.711012 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.711260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.711043 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.711260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.711110 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.711260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.711179 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpl6s\" (UniqueName: \"kubernetes.io/projected/a0da8389-92b0-42da-96cd-4dec2757e02a-kube-api-access-tpl6s\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.711260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.711229 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.711459 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.711259 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.711459 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.711338 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.711459 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.711447 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.711671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.711654 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.711770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.711745 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0da8389-92b0-42da-96cd-4dec2757e02a-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.713613 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.713588 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.713873 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.713854 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.719387 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.719363 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.719739 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.719723 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpl6s\" (UniqueName: \"kubernetes.io/projected/a0da8389-92b0-42da-96cd-4dec2757e02a-kube-api-access-tpl6s\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.783385 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.783352 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:52.812578 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.812522 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed464a2-06ae-4d29-9880-9bbf2773ac1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2418922c-mct8g\" (UID: \"aed464a2-06ae-4d29-9880-9bbf2773ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" Nov 03 20:06:52.812752 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.812734 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5hxq\" (UniqueName: \"kubernetes.io/projected/aed464a2-06ae-4d29-9880-9bbf2773ac1d-kube-api-access-n5hxq\") pod \"maas-api-revoke-sub-2418922c-mct8g\" (UID: \"aed464a2-06ae-4d29-9880-9bbf2773ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" Nov 03 20:06:52.812830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.812802 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed464a2-06ae-4d29-9880-9bbf2773ac1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2418922c-mct8g\" (UID: \"aed464a2-06ae-4d29-9880-9bbf2773ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" Nov 03 20:06:52.820567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.814359 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed464a2-06ae-4d29-9880-9bbf2773ac1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2418922c-mct8g\" (UID: \"aed464a2-06ae-4d29-9880-9bbf2773ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" Nov 03 20:06:52.820567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.816730 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed464a2-06ae-4d29-9880-9bbf2773ac1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2418922c-mct8g\" (UID: \"aed464a2-06ae-4d29-9880-9bbf2773ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" Nov 03 20:06:52.821949 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.821927 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5hxq\" (UniqueName: \"kubernetes.io/projected/aed464a2-06ae-4d29-9880-9bbf2773ac1d-kube-api-access-n5hxq\") pod \"maas-api-revoke-sub-2418922c-mct8g\" (UID: \"aed464a2-06ae-4d29-9880-9bbf2773ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" Nov 03 20:06:52.925493 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.923789 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" Nov 03 20:06:52.943198 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.943132 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5"] Nov 03 20:06:52.963187 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.962970 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5"] Nov 03 20:06:52.963592 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:52.963509 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5" Nov 03 20:06:53.035420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.035197 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286c723d-c38b-4909-b98f-9dd7b846985b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5\" (UID: \"286c723d-c38b-4909-b98f-9dd7b846985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5" Nov 03 20:06:53.035420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.035253 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz289\" (UniqueName: \"kubernetes.io/projected/286c723d-c38b-4909-b98f-9dd7b846985b-kube-api-access-tz289\") pod \"maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5\" (UID: \"286c723d-c38b-4909-b98f-9dd7b846985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5" Nov 03 20:06:53.035420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.035300 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286c723d-c38b-4909-b98f-9dd7b846985b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5\" (UID: \"286c723d-c38b-4909-b98f-9dd7b846985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5" Nov 03 20:06:53.091062 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.089577 5111 generic.go:356] "Generic (PLEG): container finished" podID="43a7ed3b-fbe3-423f-aadc-1a32f8ccad73" containerID="bd17bfa3cf7b0297a876848ab264192a85a8bffbed32547a316895a2ba46a14a" exitCode=0 Nov 03 20:06:53.091062 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.089638 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" event={"ID":"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73","Type":"ContainerDied","Data":"bd17bfa3cf7b0297a876848ab264192a85a8bffbed32547a316895a2ba46a14a"} Nov 03 20:06:53.102259 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.102167 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7"] Nov 03 20:06:53.107206 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:06:53.107030 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0da8389_92b0_42da_96cd_4dec2757e02a.slice/crio-85a680f1e565f8022dfc3dfa97844e39aa5d355345cfbe358568e3848d1c3afd WatchSource:0}: Error finding container 85a680f1e565f8022dfc3dfa97844e39aa5d355345cfbe358568e3848d1c3afd: Status 404 returned error can't find the container with id 85a680f1e565f8022dfc3dfa97844e39aa5d355345cfbe358568e3848d1c3afd Nov 03 20:06:53.111433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.110006 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:06:53.111433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.110084 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:06:53.111433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.110127 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:06:53.138182 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.138154 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz289\" (UniqueName: \"kubernetes.io/projected/286c723d-c38b-4909-b98f-9dd7b846985b-kube-api-access-tz289\") pod \"maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5\" (UID: \"286c723d-c38b-4909-b98f-9dd7b846985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5" Nov 03 20:06:53.138332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.138245 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286c723d-c38b-4909-b98f-9dd7b846985b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5\" (UID: \"286c723d-c38b-4909-b98f-9dd7b846985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5" Nov 03 20:06:53.138661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.138502 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286c723d-c38b-4909-b98f-9dd7b846985b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5\" (UID: \"286c723d-c38b-4909-b98f-9dd7b846985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5" Nov 03 20:06:53.139603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.139353 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286c723d-c38b-4909-b98f-9dd7b846985b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5\" (UID: \"286c723d-c38b-4909-b98f-9dd7b846985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5" Nov 03 20:06:53.163956 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.163912 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286c723d-c38b-4909-b98f-9dd7b846985b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5\" (UID: \"286c723d-c38b-4909-b98f-9dd7b846985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5" Nov 03 20:06:53.164517 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.164488 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz289\" (UniqueName: \"kubernetes.io/projected/286c723d-c38b-4909-b98f-9dd7b846985b-kube-api-access-tz289\") pod \"maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5\" (UID: \"286c723d-c38b-4909-b98f-9dd7b846985b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5" Nov 03 20:06:53.279861 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:06:53.279813 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed464a2_06ae_4d29_9880_9bbf2773ac1d.slice/crio-3b417f404e2449f0336996b6b88839f6adb0887a09facbc9454f94448dcbdc8a WatchSource:0}: Error finding container 3b417f404e2449f0336996b6b88839f6adb0887a09facbc9454f94448dcbdc8a: Status 404 returned error can't find the container with id 3b417f404e2449f0336996b6b88839f6adb0887a09facbc9454f94448dcbdc8a Nov 03 20:06:53.284749 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.284721 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g"] Nov 03 20:06:53.290262 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.290241 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5" Nov 03 20:06:53.602473 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:06:53.602437 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286c723d_c38b_4909_b98f_9dd7b846985b.slice/crio-747b1faa4bec889fc99f3d7aadd1698420c58a168ae5da79e18d8a77ee2bc134 WatchSource:0}: Error finding container 747b1faa4bec889fc99f3d7aadd1698420c58a168ae5da79e18d8a77ee2bc134: Status 404 returned error can't find the container with id 747b1faa4bec889fc99f3d7aadd1698420c58a168ae5da79e18d8a77ee2bc134 Nov 03 20:06:53.604941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:53.604913 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5"] Nov 03 20:06:54.103177 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.103140 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" event={"ID":"a0da8389-92b0-42da-96cd-4dec2757e02a","Type":"ContainerStarted","Data":"243bd3b731e109ef7d8ee2279d84bb5d97405998fee26922447feeed7744b67b"} Nov 03 20:06:54.103664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.103201 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" event={"ID":"a0da8389-92b0-42da-96cd-4dec2757e02a","Type":"ContainerStarted","Data":"85a680f1e565f8022dfc3dfa97844e39aa5d355345cfbe358568e3848d1c3afd"} Nov 03 20:06:54.105197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.105165 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5" event={"ID":"286c723d-c38b-4909-b98f-9dd7b846985b","Type":"ContainerStarted","Data":"747b1faa4bec889fc99f3d7aadd1698420c58a168ae5da79e18d8a77ee2bc134"} Nov 03 20:06:54.107594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.107569 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" event={"ID":"aed464a2-06ae-4d29-9880-9bbf2773ac1d","Type":"ContainerStarted","Data":"66e026ea3277d37426559c87855a5f6efdaf8984df7a8fc0391047c571c3edc9"} Nov 03 20:06:54.107707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.107615 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" event={"ID":"aed464a2-06ae-4d29-9880-9bbf2773ac1d","Type":"ContainerStarted","Data":"3b417f404e2449f0336996b6b88839f6adb0887a09facbc9454f94448dcbdc8a"} Nov 03 20:06:54.122908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.122880 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z"] Nov 03 20:06:54.128174 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.128151 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z" Nov 03 20:06:54.141841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.141492 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z"] Nov 03 20:06:54.189875 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.189555 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" podStartSLOduration=2.189511445 podStartE2EDuration="2.189511445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:54.135175381 +0000 UTC m=+1644.097553820" watchObservedRunningTime="2026-11-03 20:06:54.189511445 +0000 UTC m=+1644.151889885" Nov 03 20:06:54.197955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.194995 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed16cca-1dbc-4726-902d-0107706db272-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z\" (UID: \"7ed16cca-1dbc-4726-902d-0107706db272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z" Nov 03 20:06:54.197955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.195050 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcb4d\" (UniqueName: \"kubernetes.io/projected/7ed16cca-1dbc-4726-902d-0107706db272-kube-api-access-xcb4d\") pod \"maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z\" (UID: \"7ed16cca-1dbc-4726-902d-0107706db272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z" Nov 03 20:06:54.197955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.195289 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed16cca-1dbc-4726-902d-0107706db272-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z\" (UID: \"7ed16cca-1dbc-4726-902d-0107706db272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z" Nov 03 20:06:54.232569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.231202 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" podStartSLOduration=2.231183501 podStartE2EDuration="2.231183501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:54.225914605 +0000 UTC m=+1644.188293055" watchObservedRunningTime="2026-11-03 20:06:54.231183501 +0000 UTC m=+1644.193561943" Nov 03 20:06:54.296746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.296708 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed16cca-1dbc-4726-902d-0107706db272-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z\" (UID: \"7ed16cca-1dbc-4726-902d-0107706db272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z" Nov 03 20:06:54.297078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.297051 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcb4d\" (UniqueName: \"kubernetes.io/projected/7ed16cca-1dbc-4726-902d-0107706db272-kube-api-access-xcb4d\") pod \"maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z\" (UID: \"7ed16cca-1dbc-4726-902d-0107706db272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z" Nov 03 20:06:54.297440 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.297420 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed16cca-1dbc-4726-902d-0107706db272-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z\" (UID: \"7ed16cca-1dbc-4726-902d-0107706db272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z" Nov 03 20:06:54.298416 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.298389 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed16cca-1dbc-4726-902d-0107706db272-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z\" (UID: \"7ed16cca-1dbc-4726-902d-0107706db272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z" Nov 03 20:06:54.303734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.303455 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed16cca-1dbc-4726-902d-0107706db272-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z\" (UID: \"7ed16cca-1dbc-4726-902d-0107706db272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z" Nov 03 20:06:54.309028 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.309002 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcb4d\" (UniqueName: \"kubernetes.io/projected/7ed16cca-1dbc-4726-902d-0107706db272-kube-api-access-xcb4d\") pod \"maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z\" (UID: \"7ed16cca-1dbc-4726-902d-0107706db272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z" Nov 03 20:06:54.433078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.433051 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" Nov 03 20:06:54.501582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.500893 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-maas-api-openshift-ca\") pod \"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73\" (UID: \"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73\") " Nov 03 20:06:54.501582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.501094 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-maas-api-xks-ca\") pod \"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73\" (UID: \"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73\") " Nov 03 20:06:54.501582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.501338 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-kube-api-access-245bx\" (UniqueName: \"kubernetes.io/projected/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-kube-api-access-245bx\") pod \"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73\" (UID: \"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73\") " Nov 03 20:06:54.502972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.502939 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-maas-api-openshift-ca" pod "43a7ed3b-fbe3-423f-aadc-1a32f8ccad73" (UID: "43a7ed3b-fbe3-423f-aadc-1a32f8ccad73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:54.505351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.505330 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-maas-api-xks-ca" pod "43a7ed3b-fbe3-423f-aadc-1a32f8ccad73" (UID: "43a7ed3b-fbe3-423f-aadc-1a32f8ccad73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:54.505704 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.505681 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-kube-api-access-245bx" pod "43a7ed3b-fbe3-423f-aadc-1a32f8ccad73" (UID: "43a7ed3b-fbe3-423f-aadc-1a32f8ccad73"). InnerVolumeSpecName "kube-api-access-245bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:54.517999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.517973 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z" Nov 03 20:06:54.603868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.603839 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-245bx\" (UniqueName: \"kubernetes.io/projected/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-kube-api-access-245bx\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:54.604039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.604029 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:54.604113 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.604105 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:54.784624 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.784409 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:06:54.785961 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.785929 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:06:54.786055 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.785999 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" podUID="a0da8389-92b0-42da-96cd-4dec2757e02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 20:06:54.876591 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:06:54.876495 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed16cca_1dbc_4726_902d_0107706db272.slice/crio-7aa552f51631ebb5040b0a1217ddc13203d41be96dcc6a87649abce8c6155c95 WatchSource:0}: Error finding container 7aa552f51631ebb5040b0a1217ddc13203d41be96dcc6a87649abce8c6155c95: Status 404 returned error can't find the container with id 7aa552f51631ebb5040b0a1217ddc13203d41be96dcc6a87649abce8c6155c95 Nov 03 20:06:54.885113 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:54.885084 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z"] Nov 03 20:06:55.230612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.228709 5111 generic.go:356] "Generic (PLEG): container finished" podID="aed464a2-06ae-4d29-9880-9bbf2773ac1d" containerID="66e026ea3277d37426559c87855a5f6efdaf8984df7a8fc0391047c571c3edc9" exitCode=0 Nov 03 20:06:55.230612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.229095 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" event={"ID":"aed464a2-06ae-4d29-9880-9bbf2773ac1d","Type":"ContainerDied","Data":"66e026ea3277d37426559c87855a5f6efdaf8984df7a8fc0391047c571c3edc9"} Nov 03 20:06:55.231624 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.231590 5111 generic.go:356] "Generic (PLEG): container finished" podID="286c723d-c38b-4909-b98f-9dd7b846985b" containerID="33b34d57090caae9db412886304e8ee64463bf92cb70c9941012d7713bef3007" exitCode=0 Nov 03 20:06:55.231733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.231665 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5" event={"ID":"286c723d-c38b-4909-b98f-9dd7b846985b","Type":"ContainerDied","Data":"33b34d57090caae9db412886304e8ee64463bf92cb70c9941012d7713bef3007"} Nov 03 20:06:55.234472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.234443 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z" event={"ID":"7ed16cca-1dbc-4726-902d-0107706db272","Type":"ContainerStarted","Data":"7aa552f51631ebb5040b0a1217ddc13203d41be96dcc6a87649abce8c6155c95"} Nov 03 20:06:55.237248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.237226 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" Nov 03 20:06:55.237659 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.237631 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg" event={"ID":"43a7ed3b-fbe3-423f-aadc-1a32f8ccad73","Type":"ContainerDied","Data":"40fa7546a063852edc61f4bf524491c13980660534ea21407e093e3528f45fd0"} Nov 03 20:06:55.237811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.237798 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40fa7546a063852edc61f4bf524491c13980660534ea21407e093e3528f45fd0" Nov 03 20:06:55.599877 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.599837 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk"] Nov 03 20:06:55.600949 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.600926 5111 cpu_manager.go:414] "containerMap: removing container" podUID="43a7ed3b-fbe3-423f-aadc-1a32f8ccad73" containerName="revoke-keys" Nov 03 20:06:55.600949 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.600946 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a7ed3b-fbe3-423f-aadc-1a32f8ccad73" containerName="revoke-keys" Nov 03 20:06:55.601149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.601083 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a7ed3b-fbe3-423f-aadc-1a32f8ccad73" containerName="revoke-keys" Nov 03 20:06:55.605661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.605640 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.635990 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.635192 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk"] Nov 03 20:06:55.783876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.783841 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:06:55.784056 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.783897 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" podUID="a0da8389-92b0-42da-96cd-4dec2757e02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 20:06:55.787893 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.787861 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9562k\" (UniqueName: \"kubernetes.io/projected/407008ca-4d98-4538-9f2c-bfdd9b03535a-kube-api-access-9562k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.788100 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.788074 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.788165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.788133 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.788217 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.788195 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.788272 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.788234 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.788329 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.788289 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/407008ca-4d98-4538-9f2c-bfdd9b03535a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.788386 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.788350 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.890321 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.890239 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.890321 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.890300 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.890649 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.890326 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.890649 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.890362 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.890649 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.890387 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/407008ca-4d98-4538-9f2c-bfdd9b03535a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.890649 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.890465 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.890649 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.890621 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9562k\" (UniqueName: \"kubernetes.io/projected/407008ca-4d98-4538-9f2c-bfdd9b03535a-kube-api-access-9562k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.891171 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.891145 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.896079 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.893113 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.896182 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.895755 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/407008ca-4d98-4538-9f2c-bfdd9b03535a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.896182 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.895946 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.896287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.896264 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.899438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.899416 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9562k\" (UniqueName: \"kubernetes.io/projected/407008ca-4d98-4538-9f2c-bfdd9b03535a-kube-api-access-9562k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.899866 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.899838 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lfrxk\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:55.936849 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:55.936822 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:56.281089 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.279199 5111 generic.go:356] "Generic (PLEG): container finished" podID="7ed16cca-1dbc-4726-902d-0107706db272" containerID="8ea93620795f6f4b3b40a5ca3f8a911215de510d7a9fcbc784af36e15421143f" exitCode=0 Nov 03 20:06:56.281089 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.280100 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z" event={"ID":"7ed16cca-1dbc-4726-902d-0107706db272","Type":"ContainerDied","Data":"8ea93620795f6f4b3b40a5ca3f8a911215de510d7a9fcbc784af36e15421143f"} Nov 03 20:06:56.495603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.442641 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk"] Nov 03 20:06:56.499555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.499214 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:06:56.499555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.499327 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:06:56.700511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.700141 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5" Nov 03 20:06:56.786077 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.785566 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:06:56.786077 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.785624 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" podUID="a0da8389-92b0-42da-96cd-4dec2757e02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 20:06:56.787031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.786975 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/286c723d-c38b-4909-b98f-9dd7b846985b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286c723d-c38b-4909-b98f-9dd7b846985b-maas-api-openshift-ca\") pod \"286c723d-c38b-4909-b98f-9dd7b846985b\" (UID: \"286c723d-c38b-4909-b98f-9dd7b846985b\") " Nov 03 20:06:56.787872 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.787855 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/286c723d-c38b-4909-b98f-9dd7b846985b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286c723d-c38b-4909-b98f-9dd7b846985b-maas-api-xks-ca\") pod \"286c723d-c38b-4909-b98f-9dd7b846985b\" (UID: \"286c723d-c38b-4909-b98f-9dd7b846985b\") " Nov 03 20:06:56.788459 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.788440 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/286c723d-c38b-4909-b98f-9dd7b846985b-kube-api-access-tz289\" (UniqueName: \"kubernetes.io/projected/286c723d-c38b-4909-b98f-9dd7b846985b-kube-api-access-tz289\") pod \"286c723d-c38b-4909-b98f-9dd7b846985b\" (UID: \"286c723d-c38b-4909-b98f-9dd7b846985b\") " Nov 03 20:06:56.789969 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.787711 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/286c723d-c38b-4909-b98f-9dd7b846985b-maas-api-openshift-ca" pod "286c723d-c38b-4909-b98f-9dd7b846985b" (UID: "286c723d-c38b-4909-b98f-9dd7b846985b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:56.807568 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.802403 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/286c723d-c38b-4909-b98f-9dd7b846985b-maas-api-xks-ca" pod "286c723d-c38b-4909-b98f-9dd7b846985b" (UID: "286c723d-c38b-4909-b98f-9dd7b846985b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:56.808485 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.808444 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/286c723d-c38b-4909-b98f-9dd7b846985b-kube-api-access-tz289" pod "286c723d-c38b-4909-b98f-9dd7b846985b" (UID: "286c723d-c38b-4909-b98f-9dd7b846985b"). InnerVolumeSpecName "kube-api-access-tz289". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:56.826216 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.824902 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" Nov 03 20:06:56.894172 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.894133 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286c723d-c38b-4909-b98f-9dd7b846985b-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:56.894172 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.894174 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286c723d-c38b-4909-b98f-9dd7b846985b-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:56.894436 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.894190 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz289\" (UniqueName: \"kubernetes.io/projected/286c723d-c38b-4909-b98f-9dd7b846985b-kube-api-access-tz289\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:56.995017 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.994977 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed464a2-06ae-4d29-9880-9bbf2773ac1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed464a2-06ae-4d29-9880-9bbf2773ac1d-maas-api-openshift-ca\") pod \"aed464a2-06ae-4d29-9880-9bbf2773ac1d\" (UID: \"aed464a2-06ae-4d29-9880-9bbf2773ac1d\") " Nov 03 20:06:56.995286 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.995268 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed464a2-06ae-4d29-9880-9bbf2773ac1d-kube-api-access-n5hxq\" (UniqueName: \"kubernetes.io/projected/aed464a2-06ae-4d29-9880-9bbf2773ac1d-kube-api-access-n5hxq\") pod \"aed464a2-06ae-4d29-9880-9bbf2773ac1d\" (UID: \"aed464a2-06ae-4d29-9880-9bbf2773ac1d\") " Nov 03 20:06:56.995410 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.995359 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed464a2-06ae-4d29-9880-9bbf2773ac1d-maas-api-openshift-ca" pod "aed464a2-06ae-4d29-9880-9bbf2773ac1d" (UID: "aed464a2-06ae-4d29-9880-9bbf2773ac1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:56.995450 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.995408 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aed464a2-06ae-4d29-9880-9bbf2773ac1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed464a2-06ae-4d29-9880-9bbf2773ac1d-maas-api-xks-ca\") pod \"aed464a2-06ae-4d29-9880-9bbf2773ac1d\" (UID: \"aed464a2-06ae-4d29-9880-9bbf2773ac1d\") " Nov 03 20:06:56.996241 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.996213 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed464a2-06ae-4d29-9880-9bbf2773ac1d-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:56.997855 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.997828 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed464a2-06ae-4d29-9880-9bbf2773ac1d-kube-api-access-n5hxq" pod "aed464a2-06ae-4d29-9880-9bbf2773ac1d" (UID: "aed464a2-06ae-4d29-9880-9bbf2773ac1d"). InnerVolumeSpecName "kube-api-access-n5hxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:56.998786 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:56.998764 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed464a2-06ae-4d29-9880-9bbf2773ac1d-maas-api-xks-ca" pod "aed464a2-06ae-4d29-9880-9bbf2773ac1d" (UID: "aed464a2-06ae-4d29-9880-9bbf2773ac1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:57.099484 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.099377 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5hxq\" (UniqueName: \"kubernetes.io/projected/aed464a2-06ae-4d29-9880-9bbf2773ac1d-kube-api-access-n5hxq\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:57.099771 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.099748 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed464a2-06ae-4d29-9880-9bbf2773ac1d-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:57.306454 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.306410 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5" event={"ID":"286c723d-c38b-4909-b98f-9dd7b846985b","Type":"ContainerDied","Data":"747b1faa4bec889fc99f3d7aadd1698420c58a168ae5da79e18d8a77ee2bc134"} Nov 03 20:06:57.307109 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.306477 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="747b1faa4bec889fc99f3d7aadd1698420c58a168ae5da79e18d8a77ee2bc134" Nov 03 20:06:57.307109 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.306594 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5" Nov 03 20:06:57.315512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.315392 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" event={"ID":"407008ca-4d98-4538-9f2c-bfdd9b03535a","Type":"ContainerStarted","Data":"0c52f1466eeed8d7780f351a8661fd0c7b95439f189a7b03a7353411f5836ccf"} Nov 03 20:06:57.315512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.315466 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:57.315512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.315486 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" event={"ID":"407008ca-4d98-4538-9f2c-bfdd9b03535a","Type":"ContainerStarted","Data":"abe5d9140d56cc65e3526a8c9482dd55d1e755f2cc2157d577363d2c75d34a4c"} Nov 03 20:06:57.317443 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.317157 5111 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lfrxk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:06:57.317443 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.317210 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" podUID="407008ca-4d98-4538-9f2c-bfdd9b03535a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:06:57.323853 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.323826 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" Nov 03 20:06:57.326283 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.324103 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g" event={"ID":"aed464a2-06ae-4d29-9880-9bbf2773ac1d","Type":"ContainerDied","Data":"3b417f404e2449f0336996b6b88839f6adb0887a09facbc9454f94448dcbdc8a"} Nov 03 20:06:57.326283 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.326194 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b417f404e2449f0336996b6b88839f6adb0887a09facbc9454f94448dcbdc8a" Nov 03 20:06:57.339285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.339243 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" podStartSLOduration=2.339230775 podStartE2EDuration="2.339230775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:57.336518226 +0000 UTC m=+1647.298896666" watchObservedRunningTime="2026-11-03 20:06:57.339230775 +0000 UTC m=+1647.301609214" Nov 03 20:06:57.789787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.789703 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:06:57.791647 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.791608 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" podUID="a0da8389-92b0-42da-96cd-4dec2757e02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 20:06:57.893993 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.893962 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z" Nov 03 20:06:57.997564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.996979 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed16cca-1dbc-4726-902d-0107706db272-kube-api-access-xcb4d\" (UniqueName: \"kubernetes.io/projected/7ed16cca-1dbc-4726-902d-0107706db272-kube-api-access-xcb4d\") pod \"7ed16cca-1dbc-4726-902d-0107706db272\" (UID: \"7ed16cca-1dbc-4726-902d-0107706db272\") " Nov 03 20:06:57.997564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.997242 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ed16cca-1dbc-4726-902d-0107706db272-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed16cca-1dbc-4726-902d-0107706db272-maas-api-xks-ca\") pod \"7ed16cca-1dbc-4726-902d-0107706db272\" (UID: \"7ed16cca-1dbc-4726-902d-0107706db272\") " Nov 03 20:06:57.997564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.997343 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed16cca-1dbc-4726-902d-0107706db272-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed16cca-1dbc-4726-902d-0107706db272-maas-api-openshift-ca\") pod \"7ed16cca-1dbc-4726-902d-0107706db272\" (UID: \"7ed16cca-1dbc-4726-902d-0107706db272\") " Nov 03 20:06:57.998389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:57.998360 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed16cca-1dbc-4726-902d-0107706db272-maas-api-openshift-ca" pod "7ed16cca-1dbc-4726-902d-0107706db272" (UID: "7ed16cca-1dbc-4726-902d-0107706db272"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:58.003233 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:58.003198 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed16cca-1dbc-4726-902d-0107706db272-maas-api-xks-ca" pod "7ed16cca-1dbc-4726-902d-0107706db272" (UID: "7ed16cca-1dbc-4726-902d-0107706db272"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:58.003789 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:58.003745 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed16cca-1dbc-4726-902d-0107706db272-kube-api-access-xcb4d" pod "7ed16cca-1dbc-4726-902d-0107706db272" (UID: "7ed16cca-1dbc-4726-902d-0107706db272"). InnerVolumeSpecName "kube-api-access-xcb4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:58.099902 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:58.099788 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed16cca-1dbc-4726-902d-0107706db272-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:58.099902 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:58.099827 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcb4d\" (UniqueName: \"kubernetes.io/projected/7ed16cca-1dbc-4726-902d-0107706db272-kube-api-access-xcb4d\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:58.099902 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:58.099841 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed16cca-1dbc-4726-902d-0107706db272-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:58.337242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:58.337214 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z" Nov 03 20:06:58.337911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:58.337230 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z" event={"ID":"7ed16cca-1dbc-4726-902d-0107706db272","Type":"ContainerDied","Data":"7aa552f51631ebb5040b0a1217ddc13203d41be96dcc6a87649abce8c6155c95"} Nov 03 20:06:58.338052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:58.337940 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7aa552f51631ebb5040b0a1217ddc13203d41be96dcc6a87649abce8c6155c95" Nov 03 20:06:58.338181 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:58.338162 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:06:58.784101 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:58.784060 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:06:58.784333 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:58.784130 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" podUID="a0da8389-92b0-42da-96cd-4dec2757e02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 20:06:59.783719 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:59.783679 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:06:59.784298 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:06:59.783740 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" podUID="a0da8389-92b0-42da-96cd-4dec2757e02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 20:07:00.788824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:00.788779 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:07:00.789328 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:00.789311 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:07:00.790301 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:00.790282 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:07:00.874998 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:00.874974 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq"] Nov 03 20:07:04.244819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:04.244779 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv"] Nov 03 20:07:04.260601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:04.260563 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt"] Nov 03 20:07:05.057879 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:07:05.057840 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:07:05.875773 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:05.875681 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" podUID="9eb7c810-e130-473f-a38a-9187120e5518" containerName="payload-processing" containerID="cri-o://1ac6142633ef82e20b0c7627af57e21cffefbcc38962a1ac093e74ca8ae962d8" gracePeriod=25 Nov 03 20:07:06.328249 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:06.328212 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" Nov 03 20:07:06.423847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:06.420275 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb7c810-e130-473f-a38a-9187120e5518-kube-api-access-2hs59\" (UniqueName: \"kubernetes.io/projected/9eb7c810-e130-473f-a38a-9187120e5518-kube-api-access-2hs59\") pod \"9eb7c810-e130-473f-a38a-9187120e5518\" (UID: \"9eb7c810-e130-473f-a38a-9187120e5518\") " Nov 03 20:07:06.423847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:06.421053 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eb7c810-e130-473f-a38a-9187120e5518-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb7c810-e130-473f-a38a-9187120e5518-plugins-config-volume\") pod \"9eb7c810-e130-473f-a38a-9187120e5518\" (UID: \"9eb7c810-e130-473f-a38a-9187120e5518\") " Nov 03 20:07:06.423847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:06.422221 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb7c810-e130-473f-a38a-9187120e5518-plugins-config-volume" pod "9eb7c810-e130-473f-a38a-9187120e5518" (UID: "9eb7c810-e130-473f-a38a-9187120e5518"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:07:06.423847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:06.423751 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb7c810-e130-473f-a38a-9187120e5518-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:06.468108 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:06.468062 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb7c810-e130-473f-a38a-9187120e5518-kube-api-access-2hs59" pod "9eb7c810-e130-473f-a38a-9187120e5518" (UID: "9eb7c810-e130-473f-a38a-9187120e5518"). InnerVolumeSpecName "kube-api-access-2hs59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:06.485276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:06.485237 5111 generic.go:356] "Generic (PLEG): container finished" podID="9eb7c810-e130-473f-a38a-9187120e5518" containerID="1ac6142633ef82e20b0c7627af57e21cffefbcc38962a1ac093e74ca8ae962d8" exitCode=0 Nov 03 20:07:06.485456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:06.485336 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" Nov 03 20:07:06.485609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:06.485585 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" event={"ID":"9eb7c810-e130-473f-a38a-9187120e5518","Type":"ContainerDied","Data":"1ac6142633ef82e20b0c7627af57e21cffefbcc38962a1ac093e74ca8ae962d8"} Nov 03 20:07:06.485713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:06.485647 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq" event={"ID":"9eb7c810-e130-473f-a38a-9187120e5518","Type":"ContainerDied","Data":"38f46963912ac8cfdbe4516222ecfc79f8833608a8159e17bb3b89493683cd29"} Nov 03 20:07:06.485713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:06.485682 5111 scope.go:122] "RemoveContainer" containerID="1ac6142633ef82e20b0c7627af57e21cffefbcc38962a1ac093e74ca8ae962d8" Nov 03 20:07:06.505240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:06.505218 5111 scope.go:122] "RemoveContainer" containerID="1ac6142633ef82e20b0c7627af57e21cffefbcc38962a1ac093e74ca8ae962d8" Nov 03 20:07:06.505658 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:07:06.505631 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac6142633ef82e20b0c7627af57e21cffefbcc38962a1ac093e74ca8ae962d8\": container with ID starting with 1ac6142633ef82e20b0c7627af57e21cffefbcc38962a1ac093e74ca8ae962d8 not found: ID does not exist" containerID="1ac6142633ef82e20b0c7627af57e21cffefbcc38962a1ac093e74ca8ae962d8" Nov 03 20:07:06.505731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:06.505666 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac6142633ef82e20b0c7627af57e21cffefbcc38962a1ac093e74ca8ae962d8"} err="failed to get container status \"1ac6142633ef82e20b0c7627af57e21cffefbcc38962a1ac093e74ca8ae962d8\": rpc error: code = NotFound desc = could not find container \"1ac6142633ef82e20b0c7627af57e21cffefbcc38962a1ac093e74ca8ae962d8\": container with ID starting with 1ac6142633ef82e20b0c7627af57e21cffefbcc38962a1ac093e74ca8ae962d8 not found: ID does not exist" Nov 03 20:07:06.525851 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:06.525813 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq"] Nov 03 20:07:06.527305 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:06.527277 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hs59\" (UniqueName: \"kubernetes.io/projected/9eb7c810-e130-473f-a38a-9187120e5518-kube-api-access-2hs59\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:06.531457 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:06.531431 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8fdab3c3-59db6567f4-psqhq"] Nov 03 20:07:07.797556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:07.796630 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg"] Nov 03 20:07:07.891576 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:07.891421 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8fdab3c3-2mwmg"] Nov 03 20:07:08.458562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:08.430871 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a7ed3b-fbe3-423f-aadc-1a32f8ccad73" path="/var/lib/kubelet/pods/43a7ed3b-fbe3-423f-aadc-1a32f8ccad73/volumes" Nov 03 20:07:08.458562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:08.431741 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb7c810-e130-473f-a38a-9187120e5518" path="/var/lib/kubelet/pods/9eb7c810-e130-473f-a38a-9187120e5518/volumes" Nov 03 20:07:08.805545 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:08.805426 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:07:08.892959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:08.892915 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:08.920643 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:08.919762 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:07:09.184561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:09.180632 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:09.226564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:09.225631 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:09.248588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:09.246628 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" podUID="89b0cbea-8030-4b84-aa0f-c886f8f7b550" containerName="payload-processing" containerID="cri-o://c72896e26419e1307a1fe3055f370c7f9f427224cb7138f9d2a03b52e78d400e" gracePeriod=25 Nov 03 20:07:09.267587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:09.266637 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" podUID="20828b4f-f0b4-4ab4-95ed-646f052a2e76" containerName="payload-pre-processing" containerID="cri-o://8a562eff23e5e6911627291ccde567aad0e124f80d6d1619e5aad2dff08518c4" gracePeriod=25 Nov 03 20:07:09.297654 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:09.296645 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:09.467065 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:09.355940 5111 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:07:09.467065 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:09.355999 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" podUID="87cff690-9d53-44d6-a1dc-3f96238ca235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:09.489469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:09.489368 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:09.666571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:09.576023 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:09.666571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:09.622686 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:09.666571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:09.624575 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded" Nov 03 20:07:09.666571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:09.666236 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:09.931937 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:09.902929 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:09.976590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:09.957587 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:09.976590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:09.959025 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:07:10.043951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.005585 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:07:10.043951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.021610 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 20:07:10.043951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.023169 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:07:10.043951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.023239 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:07:10.238134 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.205243 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:10.253559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.239972 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:10.328716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.307636 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:10.497742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.497640 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:07:10.600048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.600001 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:10.676563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.676497 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:07:10.692922 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.679392 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:10.717177 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.717139 5111 generic.go:356] "Generic (PLEG): container finished" podID="20828b4f-f0b4-4ab4-95ed-646f052a2e76" containerID="8a562eff23e5e6911627291ccde567aad0e124f80d6d1619e5aad2dff08518c4" exitCode=0 Nov 03 20:07:10.802761 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.802669 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" event={"ID":"20828b4f-f0b4-4ab4-95ed-646f052a2e76","Type":"ContainerDied","Data":"8a562eff23e5e6911627291ccde567aad0e124f80d6d1619e5aad2dff08518c4"} Nov 03 20:07:10.881714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.881663 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" event={"ID":"89b0cbea-8030-4b84-aa0f-c886f8f7b550","Type":"ContainerDied","Data":"c72896e26419e1307a1fe3055f370c7f9f427224cb7138f9d2a03b52e78d400e"} Nov 03 20:07:10.882026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.882004 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:07:10.882175 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.882162 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:07:10.882325 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.882311 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:07:10.882445 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.882433 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:07:10.882600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.882586 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:07:10.882759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.882748 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:07:10.882890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.882879 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:07:10.884039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.853787 5111 generic.go:356] "Generic (PLEG): container finished" podID="89b0cbea-8030-4b84-aa0f-c886f8f7b550" containerID="c72896e26419e1307a1fe3055f370c7f9f427224cb7138f9d2a03b52e78d400e" exitCode=0 Nov 03 20:07:10.939481 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.907580 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:10.958556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.940118 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:07:11.064606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:10.960017 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:07:11.064606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:11.013619 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:07:11.064606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:11.042644 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:11.264573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:11.214507 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 20:07:11.264573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:11.215925 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:07:11.264573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:11.215972 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" podUID="75b8d7d9-7e56-4f0d-85f8-d5449b803007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:11.264573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:11.246708 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:11.330640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:11.317864 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:11.498687 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:11.498646 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:11.625893 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:11.617800 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:11.693686 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:11.693637 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:11.716894 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:11.698170 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:11.992608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:11.973345 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 20:07:12.074615 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:12.072932 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:12.220558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:12.184485 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:07:12.261614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:12.082521 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:07:12.262142 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:12.262094 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:12.287272 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:12.282853 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:12.301688 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:12.301638 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:12.350081 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:12.350035 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:07:12.504551 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:12.504486 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:12.716206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:12.716159 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:07:13.012496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:13.012452 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:07:13.196558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:13.087090 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:13.221770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:13.152511 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 20:07:13.221770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:13.173034 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:13.261559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:13.244224 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:13.261559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:13.260464 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:07:13.261559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:13.260840 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:07:13.263988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:13.263856 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:13.283137 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:13.283094 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:13.417463 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:13.374378 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:13.417463 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:13.389144 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:07:13.647162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:13.647096 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:13.716705 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:13.716646 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:13.901854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:13.901814 5111 trace.go:236] Trace[97030705]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-7cpx6" (03-Nov-2026 20:07:12.296) (total time: 1605ms): Nov 03 20:07:13.901854 ip-10-0-134-178 kubenswrapper[5111]: Trace[97030705]: [1.605446513s] [1.605446513s] END Nov 03 20:07:14.052768 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.052713 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:07:14.103565 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.103478 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6" podUID="1e292de6-b36f-481b-9d69-1961e7d8026b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 20:07:14.332157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.320680 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:14.332157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.321011 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 20:07:14.399690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.385130 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:14.399690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.385451 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:07:14.399690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.385685 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:07:14.399690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.386114 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 20:07:14.399690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.394984 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:07:14.399690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.395255 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:07:14.612930 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:07:14.594151 5111 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.411s" Nov 03 20:07:14.668807 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.668761 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:07:14.720318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.717127 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:07:14.760910 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.760877 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" Nov 03 20:07:14.782806 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.782759 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" Nov 03 20:07:14.824076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.824044 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20828b4f-f0b4-4ab4-95ed-646f052a2e76-kube-api-access-ftqrw\" (UniqueName: \"kubernetes.io/projected/20828b4f-f0b4-4ab4-95ed-646f052a2e76-kube-api-access-ftqrw\") pod \"20828b4f-f0b4-4ab4-95ed-646f052a2e76\" (UID: \"20828b4f-f0b4-4ab4-95ed-646f052a2e76\") " Nov 03 20:07:14.824552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.824513 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b0cbea-8030-4b84-aa0f-c886f8f7b550-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b0cbea-8030-4b84-aa0f-c886f8f7b550-plugins-config-volume\") pod \"89b0cbea-8030-4b84-aa0f-c886f8f7b550\" (UID: \"89b0cbea-8030-4b84-aa0f-c886f8f7b550\") " Nov 03 20:07:14.824764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.824734 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20828b4f-f0b4-4ab4-95ed-646f052a2e76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20828b4f-f0b4-4ab4-95ed-646f052a2e76-plugins-config-volume\") pod \"20828b4f-f0b4-4ab4-95ed-646f052a2e76\" (UID: \"20828b4f-f0b4-4ab4-95ed-646f052a2e76\") " Nov 03 20:07:14.824959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.824944 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b0cbea-8030-4b84-aa0f-c886f8f7b550-kube-api-access-rbljb\" (UniqueName: \"kubernetes.io/projected/89b0cbea-8030-4b84-aa0f-c886f8f7b550-kube-api-access-rbljb\") pod \"89b0cbea-8030-4b84-aa0f-c886f8f7b550\" (UID: \"89b0cbea-8030-4b84-aa0f-c886f8f7b550\") " Nov 03 20:07:14.837192 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.836845 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b0cbea-8030-4b84-aa0f-c886f8f7b550-plugins-config-volume" pod "89b0cbea-8030-4b84-aa0f-c886f8f7b550" (UID: "89b0cbea-8030-4b84-aa0f-c886f8f7b550"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:07:14.880000 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.879958 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20828b4f-f0b4-4ab4-95ed-646f052a2e76-plugins-config-volume" pod "20828b4f-f0b4-4ab4-95ed-646f052a2e76" (UID: "20828b4f-f0b4-4ab4-95ed-646f052a2e76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:07:14.907558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.897854 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:14.907558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.899345 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:14.950560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.928574 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b0cbea-8030-4b84-aa0f-c886f8f7b550-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:14.952451 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.952422 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20828b4f-f0b4-4ab4-95ed-646f052a2e76-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:14.962416 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:14.962381 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20828b4f-f0b4-4ab4-95ed-646f052a2e76-kube-api-access-ftqrw" pod "20828b4f-f0b4-4ab4-95ed-646f052a2e76" (UID: "20828b4f-f0b4-4ab4-95ed-646f052a2e76"). InnerVolumeSpecName "kube-api-access-ftqrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:15.014022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.013982 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" event={"ID":"20828b4f-f0b4-4ab4-95ed-646f052a2e76","Type":"ContainerDied","Data":"1838c9a3d7d6a34027fbd1ff95d3e8ebadbc0bcb0ae1cd2a491795bd6c1c049e"} Nov 03 20:07:15.048523 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.048473 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" event={"ID":"89b0cbea-8030-4b84-aa0f-c886f8f7b550","Type":"ContainerDied","Data":"3d52cbf5ded8da986ca9fbccac1c48697f69d6800537731377b58f629191bcde"} Nov 03 20:07:15.048913 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.048771 5111 scope.go:122] "RemoveContainer" containerID="8a562eff23e5e6911627291ccde567aad0e124f80d6d1619e5aad2dff08518c4" Nov 03 20:07:15.108622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.056195 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftqrw\" (UniqueName: \"kubernetes.io/projected/20828b4f-f0b4-4ab4-95ed-646f052a2e76-kube-api-access-ftqrw\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:15.144498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.124017 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:07:15.257376 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.257342 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b0cbea-8030-4b84-aa0f-c886f8f7b550-kube-api-access-rbljb" pod "89b0cbea-8030-4b84-aa0f-c886f8f7b550" (UID: "89b0cbea-8030-4b84-aa0f-c886f8f7b550"). InnerVolumeSpecName "kube-api-access-rbljb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:15.325558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.322978 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:07:15.326365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.326327 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:15.349831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.349799 5111 scope.go:122] "RemoveContainer" containerID="c72896e26419e1307a1fe3055f370c7f9f427224cb7138f9d2a03b52e78d400e" Nov 03 20:07:15.379264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.379226 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbljb\" (UniqueName: \"kubernetes.io/projected/89b0cbea-8030-4b84-aa0f-c886f8f7b550-kube-api-access-rbljb\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:15.390819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.390761 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:15.392446 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.392415 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:15.392718 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.392676 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 20:07:15.404315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.404221 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:07:15.430753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.392361 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:15.432365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.432332 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:15.445366 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.445325 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:15.464564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.464434 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:15.670907 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.670810 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:07:15.944617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.865489 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:07:15.997407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:15.995437 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:07:16.174666 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:16.172099 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:16.389957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:16.389908 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:16.411675 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:16.411628 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:07:16.539358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:16.537213 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 20:07:16.540635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:16.538372 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:07:16.616108 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:16.616059 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8x8xh" podUID="457e9dde-adec-4ec7-83bd-8d157b9a3362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:16.654597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:16.623676 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:16.654597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:16.633627 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:16.687374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:16.674151 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 20:07:16.701051 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:07:16.678773 5111 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20828b4f_f0b4_4ab4_95ed_646f052a2e76.slice/crio-1838c9a3d7d6a34027fbd1ff95d3e8ebadbc0bcb0ae1cd2a491795bd6c1c049e\": RecentStats: unable to find data in memory cache]" Nov 03 20:07:16.728788 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:16.718555 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:16.767556 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:07:16.706482 5111 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20828b4f_f0b4_4ab4_95ed_646f052a2e76.slice/crio-1838c9a3d7d6a34027fbd1ff95d3e8ebadbc0bcb0ae1cd2a491795bd6c1c049e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20828b4f_f0b4_4ab4_95ed_646f052a2e76.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:07:16.953761 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:16.917332 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:07:17.390734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.390690 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:17.416556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.313706 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:07:17.416556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.313806 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:07:17.442030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.441975 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:17.463641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.072640 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:17.565612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.540174 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:17.579584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.578112 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:17.591232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.581009 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 03 20:07:17.684750 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:07:17.684704 5111 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.294s" Nov 03 20:07:17.685088 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.685072 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:07:17.700627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.691101 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:07:17.702595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.701731 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:17.810611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.788621 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt" Nov 03 20:07:17.878096 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.854071 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv" Nov 03 20:07:17.881691 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.881661 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:07:17.898272 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.898246 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:07:17.898500 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.898485 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:07:17.898684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.898669 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:07:17.898820 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.898806 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:07:17.914986 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.914955 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:07:17.918668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.918640 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:07:17.995396 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:17.995369 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z"] Nov 03 20:07:18.010724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:18.010697 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea700cd3-cn94z"] Nov 03 20:07:18.021759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:18.019165 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:07:18.035078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:18.035052 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5"] Nov 03 20:07:18.101554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:18.100955 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1cca2dba-gzgx5"] Nov 03 20:07:18.205602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:18.204916 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="286c723d-c38b-4909-b98f-9dd7b846985b" path="/var/lib/kubelet/pods/286c723d-c38b-4909-b98f-9dd7b846985b/volumes" Nov 03 20:07:18.213500 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:18.205893 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed16cca-1dbc-4726-902d-0107706db272" path="/var/lib/kubelet/pods/7ed16cca-1dbc-4726-902d-0107706db272/volumes" Nov 03 20:07:18.213679 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:18.213613 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b"] Nov 03 20:07:18.229012 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:18.214076 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" podUID="746d6249-f875-423d-9efb-fa8e3b31fa63" containerName="istio-proxy" containerID="cri-o://fccefc602fa3c84e1f5d4f64b2e7f2be2c28a66938a261dc53b50ec6c346fbfd" gracePeriod=30 Nov 03 20:07:18.239614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:18.239582 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt"] Nov 03 20:07:18.261650 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:18.261582 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea700cd3-8979bd89f-g62lt"] Nov 03 20:07:18.291575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:18.280846 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv"] Nov 03 20:07:18.292523 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:18.292486 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea700cd3-5df695b668-mfrtv"] Nov 03 20:07:20.197944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:20.197699 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20828b4f-f0b4-4ab4-95ed-646f052a2e76" path="/var/lib/kubelet/pods/20828b4f-f0b4-4ab4-95ed-646f052a2e76/volumes" Nov 03 20:07:20.198467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:20.198388 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b0cbea-8030-4b84-aa0f-c886f8f7b550" path="/var/lib/kubelet/pods/89b0cbea-8030-4b84-aa0f-c886f8f7b550/volumes" Nov 03 20:07:23.943752 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:23.943703 5111 patch_prober.go:29] interesting pod/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:07:23.944397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:23.943790 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" podUID="746d6249-f875-423d-9efb-fa8e3b31fa63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.289501 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm"] Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.290389 5111 cpu_manager.go:414] "containerMap: removing container" podUID="7ed16cca-1dbc-4726-902d-0107706db272" containerName="revoke-keys" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.290405 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed16cca-1dbc-4726-902d-0107706db272" containerName="revoke-keys" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.290429 5111 cpu_manager.go:414] "containerMap: removing container" podUID="aed464a2-06ae-4d29-9880-9bbf2773ac1d" containerName="revoke-keys" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.290434 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed464a2-06ae-4d29-9880-9bbf2773ac1d" containerName="revoke-keys" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.290444 5111 cpu_manager.go:414] "containerMap: removing container" podUID="89b0cbea-8030-4b84-aa0f-c886f8f7b550" containerName="payload-processing" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.290451 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b0cbea-8030-4b84-aa0f-c886f8f7b550" containerName="payload-processing" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.290459 5111 cpu_manager.go:414] "containerMap: removing container" podUID="20828b4f-f0b4-4ab4-95ed-646f052a2e76" containerName="payload-pre-processing" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.290465 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20828b4f-f0b4-4ab4-95ed-646f052a2e76" containerName="payload-pre-processing" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.290477 5111 cpu_manager.go:414] "containerMap: removing container" podUID="286c723d-c38b-4909-b98f-9dd7b846985b" containerName="revoke-keys" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.290509 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="286c723d-c38b-4909-b98f-9dd7b846985b" containerName="revoke-keys" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.290692 5111 cpu_manager.go:414] "containerMap: removing container" podUID="9eb7c810-e130-473f-a38a-9187120e5518" containerName="payload-processing" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.290705 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb7c810-e130-473f-a38a-9187120e5518" containerName="payload-processing" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.290939 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="286c723d-c38b-4909-b98f-9dd7b846985b" containerName="revoke-keys" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.290968 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb7c810-e130-473f-a38a-9187120e5518" containerName="payload-processing" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.290981 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed16cca-1dbc-4726-902d-0107706db272" containerName="revoke-keys" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.290994 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed464a2-06ae-4d29-9880-9bbf2773ac1d" containerName="revoke-keys" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.291004 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b0cbea-8030-4b84-aa0f-c886f8f7b550" containerName="payload-processing" Nov 03 20:07:25.291553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.291012 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="20828b4f-f0b4-4ab4-95ed-646f052a2e76" containerName="payload-pre-processing" Nov 03 20:07:25.299486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.298412 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" Nov 03 20:07:25.313019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.312989 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm"] Nov 03 20:07:25.353990 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.353858 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-323cb6ed\"" Nov 03 20:07:25.353990 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.353951 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-323cb6ed-dockercfg-5ms86\"" Nov 03 20:07:25.357317 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.356666 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb478e55-23a2-4900-884f-c2cabe50a096-plugins-config-volume\") pod \"payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm\" (UID: \"bb478e55-23a2-4900-884f-c2cabe50a096\") " pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" Nov 03 20:07:25.357317 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.356750 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dnbq\" (UniqueName: \"kubernetes.io/projected/bb478e55-23a2-4900-884f-c2cabe50a096-kube-api-access-8dnbq\") pod \"payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm\" (UID: \"bb478e55-23a2-4900-884f-c2cabe50a096\") " pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" Nov 03 20:07:25.459711 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.459435 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb478e55-23a2-4900-884f-c2cabe50a096-plugins-config-volume\") pod \"payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm\" (UID: \"bb478e55-23a2-4900-884f-c2cabe50a096\") " pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" Nov 03 20:07:25.459711 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.459586 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dnbq\" (UniqueName: \"kubernetes.io/projected/bb478e55-23a2-4900-884f-c2cabe50a096-kube-api-access-8dnbq\") pod \"payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm\" (UID: \"bb478e55-23a2-4900-884f-c2cabe50a096\") " pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" Nov 03 20:07:25.461955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.460949 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb478e55-23a2-4900-884f-c2cabe50a096-plugins-config-volume\") pod \"payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm\" (UID: \"bb478e55-23a2-4900-884f-c2cabe50a096\") " pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" Nov 03 20:07:25.472489 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.472448 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dnbq\" (UniqueName: \"kubernetes.io/projected/bb478e55-23a2-4900-884f-c2cabe50a096-kube-api-access-8dnbq\") pod \"payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm\" (UID: \"bb478e55-23a2-4900-884f-c2cabe50a096\") " pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" Nov 03 20:07:25.631908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:25.631865 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" Nov 03 20:07:26.026227 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:07:26.023564 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb478e55_23a2_4900_884f_c2cabe50a096.slice/crio-934274a22b3dbe0e701711d0e1c5e95c47c483d1ca86a7b5c7f5584dfbe85789 WatchSource:0}: Error finding container 934274a22b3dbe0e701711d0e1c5e95c47c483d1ca86a7b5c7f5584dfbe85789: Status 404 returned error can't find the container with id 934274a22b3dbe0e701711d0e1c5e95c47c483d1ca86a7b5c7f5584dfbe85789 Nov 03 20:07:26.026409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.026382 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm"] Nov 03 20:07:26.391985 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.391958 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:07:26.498027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.497994 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-podinfo\") pod \"746d6249-f875-423d-9efb-fa8e3b31fa63\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " Nov 03 20:07:26.498178 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.498067 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-workload-certs\") pod \"746d6249-f875-423d-9efb-fa8e3b31fa63\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " Nov 03 20:07:26.498178 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.498132 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-workload-socket\") pod \"746d6249-f875-423d-9efb-fa8e3b31fa63\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " Nov 03 20:07:26.498178 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.498168 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-credential-socket\") pod \"746d6249-f875-423d-9efb-fa8e3b31fa63\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " Nov 03 20:07:26.498313 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.498302 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746d6249-f875-423d-9efb-fa8e3b31fa63-kube-api-access-hs56q\" (UniqueName: \"kubernetes.io/projected/746d6249-f875-423d-9efb-fa8e3b31fa63-kube-api-access-hs56q\") pod \"746d6249-f875-423d-9efb-fa8e3b31fa63\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " Nov 03 20:07:26.498403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.498384 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746d6249-f875-423d-9efb-fa8e3b31fa63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746d6249-f875-423d-9efb-fa8e3b31fa63-istiod-ca-cert\") pod \"746d6249-f875-423d-9efb-fa8e3b31fa63\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " Nov 03 20:07:26.498466 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.498390 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-workload-certs" pod "746d6249-f875-423d-9efb-fa8e3b31fa63" (UID: "746d6249-f875-423d-9efb-fa8e3b31fa63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:26.498528 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.498467 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-envoy\") pod \"746d6249-f875-423d-9efb-fa8e3b31fa63\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " Nov 03 20:07:26.498528 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.498471 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-workload-socket" pod "746d6249-f875-423d-9efb-fa8e3b31fa63" (UID: "746d6249-f875-423d-9efb-fa8e3b31fa63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:26.498528 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.498486 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-credential-socket" pod "746d6249-f875-423d-9efb-fa8e3b31fa63" (UID: "746d6249-f875-423d-9efb-fa8e3b31fa63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:26.498755 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.498656 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-token\" (UniqueName: \"kubernetes.io/projected/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-token\") pod \"746d6249-f875-423d-9efb-fa8e3b31fa63\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " Nov 03 20:07:26.498819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.498762 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-data\") pod \"746d6249-f875-423d-9efb-fa8e3b31fa63\" (UID: \"746d6249-f875-423d-9efb-fa8e3b31fa63\") " Nov 03 20:07:26.498894 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.498867 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746d6249-f875-423d-9efb-fa8e3b31fa63-istiod-ca-cert" pod "746d6249-f875-423d-9efb-fa8e3b31fa63" (UID: "746d6249-f875-423d-9efb-fa8e3b31fa63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:07:26.499077 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.499050 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-data" pod "746d6249-f875-423d-9efb-fa8e3b31fa63" (UID: "746d6249-f875-423d-9efb-fa8e3b31fa63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:26.499680 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.499650 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746d6249-f875-423d-9efb-fa8e3b31fa63-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:26.499680 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.499681 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:26.499860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.499696 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:26.499860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.499710 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:26.499860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.499723 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:26.500589 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.500557 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-podinfo" pod "746d6249-f875-423d-9efb-fa8e3b31fa63" (UID: "746d6249-f875-423d-9efb-fa8e3b31fa63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:07:26.500995 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.500969 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746d6249-f875-423d-9efb-fa8e3b31fa63-kube-api-access-hs56q" pod "746d6249-f875-423d-9efb-fa8e3b31fa63" (UID: "746d6249-f875-423d-9efb-fa8e3b31fa63"). InnerVolumeSpecName "kube-api-access-hs56q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:26.501095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.501032 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-token" pod "746d6249-f875-423d-9efb-fa8e3b31fa63" (UID: "746d6249-f875-423d-9efb-fa8e3b31fa63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:26.501141 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.501125 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-envoy" pod "746d6249-f875-423d-9efb-fa8e3b31fa63" (UID: "746d6249-f875-423d-9efb-fa8e3b31fa63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:26.601436 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.601353 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs56q\" (UniqueName: \"kubernetes.io/projected/746d6249-f875-423d-9efb-fa8e3b31fa63-kube-api-access-hs56q\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:26.601436 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.601386 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:26.601436 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.601395 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:26.601436 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:26.601407 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746d6249-f875-423d-9efb-fa8e3b31fa63-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:27.064557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:27.057557 5111 generic.go:356] "Generic (PLEG): container finished" podID="746d6249-f875-423d-9efb-fa8e3b31fa63" containerID="fccefc602fa3c84e1f5d4f64b2e7f2be2c28a66938a261dc53b50ec6c346fbfd" exitCode=0 Nov 03 20:07:27.064557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:27.057770 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" event={"ID":"746d6249-f875-423d-9efb-fa8e3b31fa63","Type":"ContainerDied","Data":"fccefc602fa3c84e1f5d4f64b2e7f2be2c28a66938a261dc53b50ec6c346fbfd"} Nov 03 20:07:27.064557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:27.057830 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" event={"ID":"746d6249-f875-423d-9efb-fa8e3b31fa63","Type":"ContainerDied","Data":"fe62c33236b91a239bd74b9264862fc547c5921d02fc46e7496330da6783a793"} Nov 03 20:07:27.064557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:27.057858 5111 scope.go:122] "RemoveContainer" containerID="fccefc602fa3c84e1f5d4f64b2e7f2be2c28a66938a261dc53b50ec6c346fbfd" Nov 03 20:07:27.064557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:27.058045 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b" Nov 03 20:07:27.101770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:27.097939 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" event={"ID":"bb478e55-23a2-4900-884f-c2cabe50a096","Type":"ContainerStarted","Data":"36e51f0aeaf3e1d7c32ae217cfd3620346759a8a0559c12df9f9782e05ce2135"} Nov 03 20:07:27.101770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:27.098011 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" event={"ID":"bb478e55-23a2-4900-884f-c2cabe50a096","Type":"ContainerStarted","Data":"934274a22b3dbe0e701711d0e1c5e95c47c483d1ca86a7b5c7f5584dfbe85789"} Nov 03 20:07:27.127555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:27.120596 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" Nov 03 20:07:27.139556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:27.136510 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" podStartSLOduration=2.136441785 podStartE2EDuration="2.136441785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:27.128326969 +0000 UTC m=+1677.090705409" watchObservedRunningTime="2026-11-03 20:07:27.136441785 +0000 UTC m=+1677.098820226" Nov 03 20:07:27.158800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:27.158764 5111 scope.go:122] "RemoveContainer" containerID="fccefc602fa3c84e1f5d4f64b2e7f2be2c28a66938a261dc53b50ec6c346fbfd" Nov 03 20:07:27.165552 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:07:27.159792 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fccefc602fa3c84e1f5d4f64b2e7f2be2c28a66938a261dc53b50ec6c346fbfd\": container with ID starting with fccefc602fa3c84e1f5d4f64b2e7f2be2c28a66938a261dc53b50ec6c346fbfd not found: ID does not exist" containerID="fccefc602fa3c84e1f5d4f64b2e7f2be2c28a66938a261dc53b50ec6c346fbfd" Nov 03 20:07:27.165552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:27.159835 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fccefc602fa3c84e1f5d4f64b2e7f2be2c28a66938a261dc53b50ec6c346fbfd"} err="failed to get container status \"fccefc602fa3c84e1f5d4f64b2e7f2be2c28a66938a261dc53b50ec6c346fbfd\": rpc error: code = NotFound desc = could not find container \"fccefc602fa3c84e1f5d4f64b2e7f2be2c28a66938a261dc53b50ec6c346fbfd\": container with ID starting with fccefc602fa3c84e1f5d4f64b2e7f2be2c28a66938a261dc53b50ec6c346fbfd not found: ID does not exist" Nov 03 20:07:27.207554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:27.205842 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b"] Nov 03 20:07:27.255557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:27.245561 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-1cca2dba-openshift-default-7fbd7c797c-c5v2b"] Nov 03 20:07:28.188846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:28.188810 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="746d6249-f875-423d-9efb-fa8e3b31fa63" path="/var/lib/kubelet/pods/746d6249-f875-423d-9efb-fa8e3b31fa63/volumes" Nov 03 20:07:35.059950 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:07:35.059908 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:07:39.023204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.023161 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w"] Nov 03 20:07:39.024579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.024517 5111 cpu_manager.go:414] "containerMap: removing container" podUID="746d6249-f875-423d-9efb-fa8e3b31fa63" containerName="istio-proxy" Nov 03 20:07:39.024579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.024557 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746d6249-f875-423d-9efb-fa8e3b31fa63" containerName="istio-proxy" Nov 03 20:07:39.024808 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.024779 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="746d6249-f875-423d-9efb-fa8e3b31fa63" containerName="istio-proxy" Nov 03 20:07:39.035755 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.035721 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" Nov 03 20:07:39.040993 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.040960 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w"] Nov 03 20:07:39.041299 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.041276 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:07:39.041455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.041334 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:07:39.041555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.041389 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:07:39.041555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.041397 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:07:39.136644 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.136607 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18889d84-a700-47a3-bd79-423d1b27b109-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w\" (UID: \"18889d84-a700-47a3-bd79-423d1b27b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" Nov 03 20:07:39.136851 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.136824 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t4mq\" (UniqueName: \"kubernetes.io/projected/18889d84-a700-47a3-bd79-423d1b27b109-kube-api-access-7t4mq\") pod \"maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w\" (UID: \"18889d84-a700-47a3-bd79-423d1b27b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" Nov 03 20:07:39.137036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.137012 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18889d84-a700-47a3-bd79-423d1b27b109-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w\" (UID: \"18889d84-a700-47a3-bd79-423d1b27b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" Nov 03 20:07:39.138073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.138055 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" Nov 03 20:07:39.241687 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.241621 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18889d84-a700-47a3-bd79-423d1b27b109-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w\" (UID: \"18889d84-a700-47a3-bd79-423d1b27b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" Nov 03 20:07:39.241901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.241844 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18889d84-a700-47a3-bd79-423d1b27b109-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w\" (UID: \"18889d84-a700-47a3-bd79-423d1b27b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" Nov 03 20:07:39.242021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.241988 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t4mq\" (UniqueName: \"kubernetes.io/projected/18889d84-a700-47a3-bd79-423d1b27b109-kube-api-access-7t4mq\") pod \"maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w\" (UID: \"18889d84-a700-47a3-bd79-423d1b27b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" Nov 03 20:07:39.243404 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.243376 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18889d84-a700-47a3-bd79-423d1b27b109-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w\" (UID: \"18889d84-a700-47a3-bd79-423d1b27b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" Nov 03 20:07:39.256828 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.246292 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18889d84-a700-47a3-bd79-423d1b27b109-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w\" (UID: \"18889d84-a700-47a3-bd79-423d1b27b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" Nov 03 20:07:39.260770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.260738 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t4mq\" (UniqueName: \"kubernetes.io/projected/18889d84-a700-47a3-bd79-423d1b27b109-kube-api-access-7t4mq\") pod \"maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w\" (UID: \"18889d84-a700-47a3-bd79-423d1b27b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" Nov 03 20:07:39.377301 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.377264 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" Nov 03 20:07:39.815594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:39.814324 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w"] Nov 03 20:07:39.822555 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:07:39.822090 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18889d84_a700_47a3_bd79_423d1b27b109.slice/crio-ef7460c90b7b51bc74eb4c512196d3b095527bdcc4ba9ebef249e0a40bcda43e WatchSource:0}: Error finding container ef7460c90b7b51bc74eb4c512196d3b095527bdcc4ba9ebef249e0a40bcda43e: Status 404 returned error can't find the container with id ef7460c90b7b51bc74eb4c512196d3b095527bdcc4ba9ebef249e0a40bcda43e Nov 03 20:07:40.346089 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:40.345991 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" event={"ID":"18889d84-a700-47a3-bd79-423d1b27b109","Type":"ContainerStarted","Data":"17ce9e6e530e9f38e21bda707c2be5704ecba34792e5c26d95e4dab8491ca38c"} Nov 03 20:07:40.346089 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:40.346063 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" event={"ID":"18889d84-a700-47a3-bd79-423d1b27b109","Type":"ContainerStarted","Data":"ef7460c90b7b51bc74eb4c512196d3b095527bdcc4ba9ebef249e0a40bcda43e"} Nov 03 20:07:40.364412 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:40.364263 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" podStartSLOduration=2.364245373 podStartE2EDuration="2.364245373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:40.362440753 +0000 UTC m=+1690.324819193" watchObservedRunningTime="2026-11-03 20:07:40.364245373 +0000 UTC m=+1690.326623814" Nov 03 20:07:41.358266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:41.358234 5111 generic.go:356] "Generic (PLEG): container finished" podID="18889d84-a700-47a3-bd79-423d1b27b109" containerID="17ce9e6e530e9f38e21bda707c2be5704ecba34792e5c26d95e4dab8491ca38c" exitCode=0 Nov 03 20:07:41.358771 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:41.358324 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" event={"ID":"18889d84-a700-47a3-bd79-423d1b27b109","Type":"ContainerDied","Data":"17ce9e6e530e9f38e21bda707c2be5704ecba34792e5c26d95e4dab8491ca38c"} Nov 03 20:07:42.032714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.032675 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg"] Nov 03 20:07:42.039183 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.039155 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b0f5cc3-zzdjg"] Nov 03 20:07:42.186076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.186042 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9423b69f-7ef4-43ed-813e-473355ff28c7" path="/var/lib/kubelet/pods/9423b69f-7ef4-43ed-813e-473355ff28c7/volumes" Nov 03 20:07:42.618862 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.618820 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" Nov 03 20:07:42.661408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.661358 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18889d84-a700-47a3-bd79-423d1b27b109-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18889d84-a700-47a3-bd79-423d1b27b109-maas-api-xks-ca\") pod \"18889d84-a700-47a3-bd79-423d1b27b109\" (UID: \"18889d84-a700-47a3-bd79-423d1b27b109\") " Nov 03 20:07:42.661627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.661503 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18889d84-a700-47a3-bd79-423d1b27b109-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18889d84-a700-47a3-bd79-423d1b27b109-maas-api-openshift-ca\") pod \"18889d84-a700-47a3-bd79-423d1b27b109\" (UID: \"18889d84-a700-47a3-bd79-423d1b27b109\") " Nov 03 20:07:42.661708 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.661651 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18889d84-a700-47a3-bd79-423d1b27b109-kube-api-access-7t4mq\" (UniqueName: \"kubernetes.io/projected/18889d84-a700-47a3-bd79-423d1b27b109-kube-api-access-7t4mq\") pod \"18889d84-a700-47a3-bd79-423d1b27b109\" (UID: \"18889d84-a700-47a3-bd79-423d1b27b109\") " Nov 03 20:07:42.662095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.662056 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18889d84-a700-47a3-bd79-423d1b27b109-maas-api-openshift-ca" pod "18889d84-a700-47a3-bd79-423d1b27b109" (UID: "18889d84-a700-47a3-bd79-423d1b27b109"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:07:42.662449 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.662425 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18889d84-a700-47a3-bd79-423d1b27b109-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:42.663998 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.663972 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18889d84-a700-47a3-bd79-423d1b27b109-kube-api-access-7t4mq" pod "18889d84-a700-47a3-bd79-423d1b27b109" (UID: "18889d84-a700-47a3-bd79-423d1b27b109"). InnerVolumeSpecName "kube-api-access-7t4mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:42.664204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.664182 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18889d84-a700-47a3-bd79-423d1b27b109-maas-api-xks-ca" pod "18889d84-a700-47a3-bd79-423d1b27b109" (UID: "18889d84-a700-47a3-bd79-423d1b27b109"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:07:42.722969 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.722935 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc"] Nov 03 20:07:42.723773 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.723753 5111 cpu_manager.go:414] "containerMap: removing container" podUID="18889d84-a700-47a3-bd79-423d1b27b109" containerName="revoke-keys" Nov 03 20:07:42.723846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.723774 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18889d84-a700-47a3-bd79-423d1b27b109" containerName="revoke-keys" Nov 03 20:07:42.723944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.723933 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="18889d84-a700-47a3-bd79-423d1b27b109" containerName="revoke-keys" Nov 03 20:07:42.729662 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.728852 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.745513 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.743341 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-944e83-openshift-default-dockercfg-gj745\"" Nov 03 20:07:42.765206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.764239 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-workload-certs\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.765206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.764328 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-podinfo\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.766570 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.766519 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-token\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.766734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.766669 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-data\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.766809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.766777 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9vmd\" (UniqueName: \"kubernetes.io/projected/53ba0307-0ce6-43c8-8599-2fb0cd354f61-kube-api-access-n9vmd\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.766869 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.766847 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-envoy\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.766927 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.766888 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-credential-socket\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.767021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.767003 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-workload-socket\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.767085 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.767048 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istiod-ca-cert\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.767304 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.767285 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t4mq\" (UniqueName: \"kubernetes.io/projected/18889d84-a700-47a3-bd79-423d1b27b109-kube-api-access-7t4mq\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:42.767383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.767316 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18889d84-a700-47a3-bd79-423d1b27b109-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:42.777406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.777369 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc"] Nov 03 20:07:42.868524 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.868477 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-token\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.868734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.868631 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-data\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.868734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.868726 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9vmd\" (UniqueName: \"kubernetes.io/projected/53ba0307-0ce6-43c8-8599-2fb0cd354f61-kube-api-access-n9vmd\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.868824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.868789 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-envoy\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.868860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.868828 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-credential-socket\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.869435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.868932 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-workload-socket\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.869435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.868967 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istiod-ca-cert\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.869435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.869058 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-workload-certs\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.869435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.869111 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-podinfo\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.872890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.872851 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-data\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.873396 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.873365 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-podinfo\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.873733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.873707 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-workload-socket\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.874006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.873975 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-credential-socket\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.875990 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.875936 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istiod-ca-cert\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.877645 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.877606 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-workload-certs\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.900898 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.900859 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-envoy\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.926556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.920101 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-token\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:42.932557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:42.928097 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9vmd\" (UniqueName: \"kubernetes.io/projected/53ba0307-0ce6-43c8-8599-2fb0cd354f61-kube-api-access-n9vmd\") pod \"e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:43.077340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:43.077276 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:43.378071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:43.378040 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" Nov 03 20:07:43.378263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:43.378070 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w" event={"ID":"18889d84-a700-47a3-bd79-423d1b27b109","Type":"ContainerDied","Data":"ef7460c90b7b51bc74eb4c512196d3b095527bdcc4ba9ebef249e0a40bcda43e"} Nov 03 20:07:43.378263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:43.378123 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef7460c90b7b51bc74eb4c512196d3b095527bdcc4ba9ebef249e0a40bcda43e" Nov 03 20:07:43.426200 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:07:43.426160 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ba0307_0ce6_43c8_8599_2fb0cd354f61.slice/crio-371889d8983bba7cf5267237e7c569a5a1b6222ebddfe7e349fe1c8cffc2ccd7 WatchSource:0}: Error finding container 371889d8983bba7cf5267237e7c569a5a1b6222ebddfe7e349fe1c8cffc2ccd7: Status 404 returned error can't find the container with id 371889d8983bba7cf5267237e7c569a5a1b6222ebddfe7e349fe1c8cffc2ccd7 Nov 03 20:07:43.456557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:43.450263 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc"] Nov 03 20:07:43.472411 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:43.460247 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:07:43.472411 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:43.460350 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:07:43.472411 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:43.460398 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:07:44.223883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.223843 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss"] Nov 03 20:07:44.229140 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.229109 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss" Nov 03 20:07:44.238507 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.238472 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss"] Nov 03 20:07:44.240225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.239727 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:07:44.240225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.239731 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:07:44.240225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.239847 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:07:44.240225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.239938 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:07:44.307876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.307827 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4651e6-ec76-410a-a6fe-a874cf7a5712-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-323cb6ed-wztss\" (UID: \"5f4651e6-ec76-410a-a6fe-a874cf7a5712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss" Nov 03 20:07:44.308069 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.307968 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4651e6-ec76-410a-a6fe-a874cf7a5712-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-323cb6ed-wztss\" (UID: \"5f4651e6-ec76-410a-a6fe-a874cf7a5712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss" Nov 03 20:07:44.308640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.308303 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g5lv\" (UniqueName: \"kubernetes.io/projected/5f4651e6-ec76-410a-a6fe-a874cf7a5712-kube-api-access-2g5lv\") pod \"maas-api-revoke-keys-e2e-ait-323cb6ed-wztss\" (UID: \"5f4651e6-ec76-410a-a6fe-a874cf7a5712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss" Nov 03 20:07:44.390661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.390621 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" event={"ID":"53ba0307-0ce6-43c8-8599-2fb0cd354f61","Type":"ContainerStarted","Data":"530c601459beaf49d2879234606910984cf2364773161ab50d0305ce6635ecfa"} Nov 03 20:07:44.390865 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.390686 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" event={"ID":"53ba0307-0ce6-43c8-8599-2fb0cd354f61","Type":"ContainerStarted","Data":"371889d8983bba7cf5267237e7c569a5a1b6222ebddfe7e349fe1c8cffc2ccd7"} Nov 03 20:07:44.410315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.410258 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4651e6-ec76-410a-a6fe-a874cf7a5712-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-323cb6ed-wztss\" (UID: \"5f4651e6-ec76-410a-a6fe-a874cf7a5712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss" Nov 03 20:07:44.410525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.410364 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4651e6-ec76-410a-a6fe-a874cf7a5712-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-323cb6ed-wztss\" (UID: \"5f4651e6-ec76-410a-a6fe-a874cf7a5712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss" Nov 03 20:07:44.410625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.410605 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g5lv\" (UniqueName: \"kubernetes.io/projected/5f4651e6-ec76-410a-a6fe-a874cf7a5712-kube-api-access-2g5lv\") pod \"maas-api-revoke-keys-e2e-ait-323cb6ed-wztss\" (UID: \"5f4651e6-ec76-410a-a6fe-a874cf7a5712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss" Nov 03 20:07:44.411242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.411185 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" podStartSLOduration=2.411167837 podStartE2EDuration="2.411167837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:44.409280639 +0000 UTC m=+1694.371659080" watchObservedRunningTime="2026-11-03 20:07:44.411167837 +0000 UTC m=+1694.373546278" Nov 03 20:07:44.411510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.411483 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4651e6-ec76-410a-a6fe-a874cf7a5712-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-323cb6ed-wztss\" (UID: \"5f4651e6-ec76-410a-a6fe-a874cf7a5712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss" Nov 03 20:07:44.413561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.413523 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4651e6-ec76-410a-a6fe-a874cf7a5712-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-323cb6ed-wztss\" (UID: \"5f4651e6-ec76-410a-a6fe-a874cf7a5712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss" Nov 03 20:07:44.422077 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.422039 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g5lv\" (UniqueName: \"kubernetes.io/projected/5f4651e6-ec76-410a-a6fe-a874cf7a5712-kube-api-access-2g5lv\") pod \"maas-api-revoke-keys-e2e-ait-323cb6ed-wztss\" (UID: \"5f4651e6-ec76-410a-a6fe-a874cf7a5712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss" Nov 03 20:07:44.551672 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.550325 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss" Nov 03 20:07:44.916210 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:07:44.916178 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4651e6_ec76_410a_a6fe_a874cf7a5712.slice/crio-66e0a22aa9db7077850f2b1947d4d06813ed8784f3d34bbd814ae997c1815030 WatchSource:0}: Error finding container 66e0a22aa9db7077850f2b1947d4d06813ed8784f3d34bbd814ae997c1815030: Status 404 returned error can't find the container with id 66e0a22aa9db7077850f2b1947d4d06813ed8784f3d34bbd814ae997c1815030 Nov 03 20:07:44.921141 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:44.921107 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss"] Nov 03 20:07:45.078174 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:45.078128 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:45.084910 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:45.084875 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:45.402882 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:45.402824 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss" event={"ID":"5f4651e6-ec76-410a-a6fe-a874cf7a5712","Type":"ContainerStarted","Data":"66e0a22aa9db7077850f2b1947d4d06813ed8784f3d34bbd814ae997c1815030"} Nov 03 20:07:45.403409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:45.403078 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:45.404267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:45.404242 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:07:46.415102 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:46.415056 5111 generic.go:356] "Generic (PLEG): container finished" podID="5f4651e6-ec76-410a-a6fe-a874cf7a5712" containerID="f7558f704537297532143c6030d2402c1c59a77f18cf2416e48c2fb83954a280" exitCode=0 Nov 03 20:07:46.415642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:46.415149 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss" event={"ID":"5f4651e6-ec76-410a-a6fe-a874cf7a5712","Type":"ContainerDied","Data":"f7558f704537297532143c6030d2402c1c59a77f18cf2416e48c2fb83954a280"} Nov 03 20:07:47.801908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:47.801881 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss" Nov 03 20:07:47.956673 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:47.956629 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f4651e6-ec76-410a-a6fe-a874cf7a5712-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4651e6-ec76-410a-a6fe-a874cf7a5712-maas-api-xks-ca\") pod \"5f4651e6-ec76-410a-a6fe-a874cf7a5712\" (UID: \"5f4651e6-ec76-410a-a6fe-a874cf7a5712\") " Nov 03 20:07:47.956943 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:47.956908 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f4651e6-ec76-410a-a6fe-a874cf7a5712-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4651e6-ec76-410a-a6fe-a874cf7a5712-maas-api-openshift-ca\") pod \"5f4651e6-ec76-410a-a6fe-a874cf7a5712\" (UID: \"5f4651e6-ec76-410a-a6fe-a874cf7a5712\") " Nov 03 20:07:47.957092 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:47.957013 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4651e6-ec76-410a-a6fe-a874cf7a5712-kube-api-access-2g5lv\" (UniqueName: \"kubernetes.io/projected/5f4651e6-ec76-410a-a6fe-a874cf7a5712-kube-api-access-2g5lv\") pod \"5f4651e6-ec76-410a-a6fe-a874cf7a5712\" (UID: \"5f4651e6-ec76-410a-a6fe-a874cf7a5712\") " Nov 03 20:07:47.957393 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:47.957369 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4651e6-ec76-410a-a6fe-a874cf7a5712-maas-api-openshift-ca" pod "5f4651e6-ec76-410a-a6fe-a874cf7a5712" (UID: "5f4651e6-ec76-410a-a6fe-a874cf7a5712"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:07:47.958210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:47.958183 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4651e6-ec76-410a-a6fe-a874cf7a5712-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:47.961602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:47.961266 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4651e6-ec76-410a-a6fe-a874cf7a5712-kube-api-access-2g5lv" pod "5f4651e6-ec76-410a-a6fe-a874cf7a5712" (UID: "5f4651e6-ec76-410a-a6fe-a874cf7a5712"). InnerVolumeSpecName "kube-api-access-2g5lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:47.965858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:47.965831 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f4651e6-ec76-410a-a6fe-a874cf7a5712-maas-api-xks-ca" pod "5f4651e6-ec76-410a-a6fe-a874cf7a5712" (UID: "5f4651e6-ec76-410a-a6fe-a874cf7a5712"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:07:48.026918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.026815 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq"] Nov 03 20:07:48.028291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.028259 5111 cpu_manager.go:414] "containerMap: removing container" podUID="5f4651e6-ec76-410a-a6fe-a874cf7a5712" containerName="revoke-keys" Nov 03 20:07:48.028455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.028442 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4651e6-ec76-410a-a6fe-a874cf7a5712" containerName="revoke-keys" Nov 03 20:07:48.028877 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.028857 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4651e6-ec76-410a-a6fe-a874cf7a5712" containerName="revoke-keys" Nov 03 20:07:48.042232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.042196 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" Nov 03 20:07:48.042784 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.042234 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq"] Nov 03 20:07:48.046417 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.046393 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-944e83-dockercfg-ds6nr\"" Nov 03 20:07:48.047068 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.047012 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-944e83\"" Nov 03 20:07:48.063045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.062101 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g5lv\" (UniqueName: \"kubernetes.io/projected/5f4651e6-ec76-410a-a6fe-a874cf7a5712-kube-api-access-2g5lv\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:48.063045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.062136 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4651e6-ec76-410a-a6fe-a874cf7a5712-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:48.165555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.165147 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c80ce62-7273-46ef-ba44-b9ced07ddfe2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq\" (UID: \"2c80ce62-7273-46ef-ba44-b9ced07ddfe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" Nov 03 20:07:48.165555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.165324 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gblml\" (UniqueName: \"kubernetes.io/projected/2c80ce62-7273-46ef-ba44-b9ced07ddfe2-kube-api-access-gblml\") pod \"payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq\" (UID: \"2c80ce62-7273-46ef-ba44-b9ced07ddfe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" Nov 03 20:07:48.325611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.312654 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c80ce62-7273-46ef-ba44-b9ced07ddfe2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq\" (UID: \"2c80ce62-7273-46ef-ba44-b9ced07ddfe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" Nov 03 20:07:48.325611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.312854 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gblml\" (UniqueName: \"kubernetes.io/projected/2c80ce62-7273-46ef-ba44-b9ced07ddfe2-kube-api-access-gblml\") pod \"payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq\" (UID: \"2c80ce62-7273-46ef-ba44-b9ced07ddfe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" Nov 03 20:07:48.325611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.314343 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c80ce62-7273-46ef-ba44-b9ced07ddfe2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq\" (UID: \"2c80ce62-7273-46ef-ba44-b9ced07ddfe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" Nov 03 20:07:48.408553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.408024 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gblml\" (UniqueName: \"kubernetes.io/projected/2c80ce62-7273-46ef-ba44-b9ced07ddfe2-kube-api-access-gblml\") pod \"payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq\" (UID: \"2c80ce62-7273-46ef-ba44-b9ced07ddfe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" Nov 03 20:07:48.491828 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.491793 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss" Nov 03 20:07:48.492012 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.491876 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss" event={"ID":"5f4651e6-ec76-410a-a6fe-a874cf7a5712","Type":"ContainerDied","Data":"66e0a22aa9db7077850f2b1947d4d06813ed8784f3d34bbd814ae997c1815030"} Nov 03 20:07:48.492012 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.491938 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66e0a22aa9db7077850f2b1947d4d06813ed8784f3d34bbd814ae997c1815030" Nov 03 20:07:48.671457 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:48.671417 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" Nov 03 20:07:49.042058 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:49.039410 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq"] Nov 03 20:07:49.577098 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:49.576329 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" event={"ID":"2c80ce62-7273-46ef-ba44-b9ced07ddfe2","Type":"ContainerStarted","Data":"e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8"} Nov 03 20:07:49.577098 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:49.576404 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" event={"ID":"2c80ce62-7273-46ef-ba44-b9ced07ddfe2","Type":"ContainerStarted","Data":"1efddefede5de86bb27930e18e1d00ecbf2f70762d127318b1070e2315bbd08e"} Nov 03 20:07:49.577098 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:49.576779 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" Nov 03 20:07:49.608225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:49.598103 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" podStartSLOduration=2.598080143 podStartE2EDuration="2.598080143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:49.59248983 +0000 UTC m=+1699.554868269" watchObservedRunningTime="2026-11-03 20:07:49.598080143 +0000 UTC m=+1699.560458585" Nov 03 20:07:51.724558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:51.720633 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:51.727652 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:51.727609 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:07:51.863586 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:51.770646 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:52.033786 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.033685 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk"] Nov 03 20:07:52.231206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.231147 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:52.231465 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.066074 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:52.317661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.317515 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 20:07:52.326012 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.064159 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:52.337856 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.334196 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:52.549826 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.549777 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:07:52.777902 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.736855 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:52.827559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.798304 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:52.827559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.798502 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:52.911133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.870974 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk"] Nov 03 20:07:52.911133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.871104 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:07:52.911133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.881181 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" Nov 03 20:07:52.911133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.886013 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:07:52.911133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.886192 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:07:52.911133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.886270 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:07:52.911133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.886350 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:07:52.911133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.904034 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:07:52.961908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.961876 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:07:52.962230 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.962207 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:07:52.962400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.962387 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:07:52.962648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.962631 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:07:52.973394 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.973360 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": context deadline exceeded" Nov 03 20:07:52.988450 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:52.976708 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:07:53.002468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:53.002422 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd0382f-8e90-4ce8-b2de-7d710be227f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d70263d-5fksk\" (UID: \"cfd0382f-8e90-4ce8-b2de-7d710be227f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" Nov 03 20:07:53.002846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:53.002813 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd0382f-8e90-4ce8-b2de-7d710be227f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d70263d-5fksk\" (UID: \"cfd0382f-8e90-4ce8-b2de-7d710be227f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" Nov 03 20:07:53.004485 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:53.004452 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2bsd\" (UniqueName: \"kubernetes.io/projected/cfd0382f-8e90-4ce8-b2de-7d710be227f5-kube-api-access-s2bsd\") pod \"maas-api-revoke-sub-5d70263d-5fksk\" (UID: \"cfd0382f-8e90-4ce8-b2de-7d710be227f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" Nov 03 20:07:53.071319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:53.071236 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf"] Nov 03 20:07:53.087085 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:53.087052 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf58a55-wbhcf"] Nov 03 20:07:53.105909 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:53.105865 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd0382f-8e90-4ce8-b2de-7d710be227f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d70263d-5fksk\" (UID: \"cfd0382f-8e90-4ce8-b2de-7d710be227f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" Nov 03 20:07:53.107964 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:53.107935 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2bsd\" (UniqueName: \"kubernetes.io/projected/cfd0382f-8e90-4ce8-b2de-7d710be227f5-kube-api-access-s2bsd\") pod \"maas-api-revoke-sub-5d70263d-5fksk\" (UID: \"cfd0382f-8e90-4ce8-b2de-7d710be227f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" Nov 03 20:07:53.108227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:53.108209 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd0382f-8e90-4ce8-b2de-7d710be227f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d70263d-5fksk\" (UID: \"cfd0382f-8e90-4ce8-b2de-7d710be227f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" Nov 03 20:07:53.109176 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:53.109152 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd0382f-8e90-4ce8-b2de-7d710be227f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d70263d-5fksk\" (UID: \"cfd0382f-8e90-4ce8-b2de-7d710be227f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" Nov 03 20:07:53.111230 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:53.109881 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd0382f-8e90-4ce8-b2de-7d710be227f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d70263d-5fksk\" (UID: \"cfd0382f-8e90-4ce8-b2de-7d710be227f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" Nov 03 20:07:53.120078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:53.120054 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2bsd\" (UniqueName: \"kubernetes.io/projected/cfd0382f-8e90-4ce8-b2de-7d710be227f5-kube-api-access-s2bsd\") pod \"maas-api-revoke-sub-5d70263d-5fksk\" (UID: \"cfd0382f-8e90-4ce8-b2de-7d710be227f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" Nov 03 20:07:53.355618 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:53.355520 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" Nov 03 20:07:53.663895 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:07:53.663853 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd0382f_8e90_4ce8_b2de_7d710be227f5.slice/crio-2547f4ea53d9377e454d7479cc9c654b21e9aadb351807677a6a9bbf4d6489bd WatchSource:0}: Error finding container 2547f4ea53d9377e454d7479cc9c654b21e9aadb351807677a6a9bbf4d6489bd: Status 404 returned error can't find the container with id 2547f4ea53d9377e454d7479cc9c654b21e9aadb351807677a6a9bbf4d6489bd Nov 03 20:07:53.665730 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:53.665699 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk"] Nov 03 20:07:54.030735 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:54.030651 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2"] Nov 03 20:07:54.037114 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:54.037080 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3490817-hshq2"] Nov 03 20:07:54.185379 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:54.185346 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc9e40c-d149-44a0-a0e6-c7081d54e530" path="/var/lib/kubelet/pods/9fc9e40c-d149-44a0-a0e6-c7081d54e530/volumes" Nov 03 20:07:54.185919 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:54.185903 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3583b36-503b-4421-a11a-8ea6e61bd31f" path="/var/lib/kubelet/pods/b3583b36-503b-4421-a11a-8ea6e61bd31f/volumes" Nov 03 20:07:54.262882 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:54.262837 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" event={"ID":"cfd0382f-8e90-4ce8-b2de-7d710be227f5","Type":"ContainerStarted","Data":"e3e792cc40043536a8d9e353f33a87dd2190f3cb9e9ce615357b9556915659d5"} Nov 03 20:07:54.263043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:54.262911 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" event={"ID":"cfd0382f-8e90-4ce8-b2de-7d710be227f5","Type":"ContainerStarted","Data":"2547f4ea53d9377e454d7479cc9c654b21e9aadb351807677a6a9bbf4d6489bd"} Nov 03 20:07:54.288448 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:54.288299 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" podStartSLOduration=3.28828228 podStartE2EDuration="3.28828228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:54.287031347 +0000 UTC m=+1704.249409783" watchObservedRunningTime="2026-11-03 20:07:54.28828228 +0000 UTC m=+1704.250660720" Nov 03 20:07:54.422080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:54.422045 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm"] Nov 03 20:07:55.277726 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:55.277687 5111 generic.go:356] "Generic (PLEG): container finished" podID="cfd0382f-8e90-4ce8-b2de-7d710be227f5" containerID="e3e792cc40043536a8d9e353f33a87dd2190f3cb9e9ce615357b9556915659d5" exitCode=0 Nov 03 20:07:55.278273 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:55.277767 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" event={"ID":"cfd0382f-8e90-4ce8-b2de-7d710be227f5","Type":"ContainerDied","Data":"e3e792cc40043536a8d9e353f33a87dd2190f3cb9e9ce615357b9556915659d5"} Nov 03 20:07:56.027013 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:56.026965 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv"] Nov 03 20:07:56.034437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:56.034402 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fa764a6-tlgbv"] Nov 03 20:07:56.187063 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:56.187033 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cabe4bba-89d1-46b0-8ced-a8cdb34383e2" path="/var/lib/kubelet/pods/cabe4bba-89d1-46b0-8ced-a8cdb34383e2/volumes" Nov 03 20:07:56.497636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:56.497610 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" Nov 03 20:07:56.597100 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:56.597011 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd0382f-8e90-4ce8-b2de-7d710be227f5-kube-api-access-s2bsd\" (UniqueName: \"kubernetes.io/projected/cfd0382f-8e90-4ce8-b2de-7d710be227f5-kube-api-access-s2bsd\") pod \"cfd0382f-8e90-4ce8-b2de-7d710be227f5\" (UID: \"cfd0382f-8e90-4ce8-b2de-7d710be227f5\") " Nov 03 20:07:56.597279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:56.597106 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfd0382f-8e90-4ce8-b2de-7d710be227f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd0382f-8e90-4ce8-b2de-7d710be227f5-maas-api-openshift-ca\") pod \"cfd0382f-8e90-4ce8-b2de-7d710be227f5\" (UID: \"cfd0382f-8e90-4ce8-b2de-7d710be227f5\") " Nov 03 20:07:56.597420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:56.597389 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfd0382f-8e90-4ce8-b2de-7d710be227f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd0382f-8e90-4ce8-b2de-7d710be227f5-maas-api-xks-ca\") pod \"cfd0382f-8e90-4ce8-b2de-7d710be227f5\" (UID: \"cfd0382f-8e90-4ce8-b2de-7d710be227f5\") " Nov 03 20:07:56.597614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:56.597509 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd0382f-8e90-4ce8-b2de-7d710be227f5-maas-api-openshift-ca" pod "cfd0382f-8e90-4ce8-b2de-7d710be227f5" (UID: "cfd0382f-8e90-4ce8-b2de-7d710be227f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:07:56.598068 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:56.598049 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd0382f-8e90-4ce8-b2de-7d710be227f5-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:56.599441 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:56.599412 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd0382f-8e90-4ce8-b2de-7d710be227f5-kube-api-access-s2bsd" pod "cfd0382f-8e90-4ce8-b2de-7d710be227f5" (UID: "cfd0382f-8e90-4ce8-b2de-7d710be227f5"). InnerVolumeSpecName "kube-api-access-s2bsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:56.599522 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:56.599490 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd0382f-8e90-4ce8-b2de-7d710be227f5-maas-api-xks-ca" pod "cfd0382f-8e90-4ce8-b2de-7d710be227f5" (UID: "cfd0382f-8e90-4ce8-b2de-7d710be227f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:07:56.699455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:56.699421 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd0382f-8e90-4ce8-b2de-7d710be227f5-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:56.699455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:56.699450 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2bsd\" (UniqueName: \"kubernetes.io/projected/cfd0382f-8e90-4ce8-b2de-7d710be227f5-kube-api-access-s2bsd\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:57.308006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:57.307970 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" Nov 03 20:07:57.308006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:57.307989 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk" event={"ID":"cfd0382f-8e90-4ce8-b2de-7d710be227f5","Type":"ContainerDied","Data":"2547f4ea53d9377e454d7479cc9c654b21e9aadb351807677a6a9bbf4d6489bd"} Nov 03 20:07:57.308294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:57.308060 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2547f4ea53d9377e454d7479cc9c654b21e9aadb351807677a6a9bbf4d6489bd" Nov 03 20:07:57.384862 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:57.384822 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w"] Nov 03 20:07:57.391596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:57.391408 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c13f7966-76p4w"] Nov 03 20:07:58.043247 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:58.043208 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh"] Nov 03 20:07:58.051087 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:58.051047 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f95ec60-sp9kh"] Nov 03 20:07:58.201132 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:58.201092 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18889d84-a700-47a3-bd79-423d1b27b109" path="/var/lib/kubelet/pods/18889d84-a700-47a3-bd79-423d1b27b109/volumes" Nov 03 20:07:58.201750 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:58.201732 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6427e57d-fa48-49c8-a6a4-dca0994f967a" path="/var/lib/kubelet/pods/6427e57d-fa48-49c8-a6a4-dca0994f967a/volumes" Nov 03 20:07:59.422896 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:59.422849 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" podUID="bb478e55-23a2-4900-884f-c2cabe50a096" containerName="payload-processing" containerID="cri-o://36e51f0aeaf3e1d7c32ae217cfd3620346759a8a0559c12df9f9782e05ce2135" gracePeriod=25 Nov 03 20:07:59.750705 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:59.750675 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" Nov 03 20:07:59.864839 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:59.864798 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb478e55-23a2-4900-884f-c2cabe50a096-kube-api-access-8dnbq\" (UniqueName: \"kubernetes.io/projected/bb478e55-23a2-4900-884f-c2cabe50a096-kube-api-access-8dnbq\") pod \"bb478e55-23a2-4900-884f-c2cabe50a096\" (UID: \"bb478e55-23a2-4900-884f-c2cabe50a096\") " Nov 03 20:07:59.865017 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:59.864907 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb478e55-23a2-4900-884f-c2cabe50a096-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb478e55-23a2-4900-884f-c2cabe50a096-plugins-config-volume\") pod \"bb478e55-23a2-4900-884f-c2cabe50a096\" (UID: \"bb478e55-23a2-4900-884f-c2cabe50a096\") " Nov 03 20:07:59.865370 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:59.865339 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb478e55-23a2-4900-884f-c2cabe50a096-plugins-config-volume" pod "bb478e55-23a2-4900-884f-c2cabe50a096" (UID: "bb478e55-23a2-4900-884f-c2cabe50a096"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:07:59.865752 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:59.865735 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb478e55-23a2-4900-884f-c2cabe50a096-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:59.867236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:59.867208 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb478e55-23a2-4900-884f-c2cabe50a096-kube-api-access-8dnbq" pod "bb478e55-23a2-4900-884f-c2cabe50a096" (UID: "bb478e55-23a2-4900-884f-c2cabe50a096"). InnerVolumeSpecName "kube-api-access-8dnbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:59.967069 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:07:59.966978 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dnbq\" (UniqueName: \"kubernetes.io/projected/bb478e55-23a2-4900-884f-c2cabe50a096-kube-api-access-8dnbq\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:00.352130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:00.352095 5111 generic.go:356] "Generic (PLEG): container finished" podID="bb478e55-23a2-4900-884f-c2cabe50a096" containerID="36e51f0aeaf3e1d7c32ae217cfd3620346759a8a0559c12df9f9782e05ce2135" exitCode=0 Nov 03 20:08:00.352392 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:00.352368 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" event={"ID":"bb478e55-23a2-4900-884f-c2cabe50a096","Type":"ContainerDied","Data":"36e51f0aeaf3e1d7c32ae217cfd3620346759a8a0559c12df9f9782e05ce2135"} Nov 03 20:08:00.352507 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:00.352425 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" event={"ID":"bb478e55-23a2-4900-884f-c2cabe50a096","Type":"ContainerDied","Data":"934274a22b3dbe0e701711d0e1c5e95c47c483d1ca86a7b5c7f5584dfbe85789"} Nov 03 20:08:00.352507 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:00.352456 5111 scope.go:122] "RemoveContainer" containerID="36e51f0aeaf3e1d7c32ae217cfd3620346759a8a0559c12df9f9782e05ce2135" Nov 03 20:08:00.352650 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:00.352632 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm" Nov 03 20:08:00.388883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:00.388852 5111 scope.go:122] "RemoveContainer" containerID="36e51f0aeaf3e1d7c32ae217cfd3620346759a8a0559c12df9f9782e05ce2135" Nov 03 20:08:00.393286 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:08:00.393012 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36e51f0aeaf3e1d7c32ae217cfd3620346759a8a0559c12df9f9782e05ce2135\": container with ID starting with 36e51f0aeaf3e1d7c32ae217cfd3620346759a8a0559c12df9f9782e05ce2135 not found: ID does not exist" containerID="36e51f0aeaf3e1d7c32ae217cfd3620346759a8a0559c12df9f9782e05ce2135" Nov 03 20:08:00.393286 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:00.393058 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e51f0aeaf3e1d7c32ae217cfd3620346759a8a0559c12df9f9782e05ce2135"} err="failed to get container status \"36e51f0aeaf3e1d7c32ae217cfd3620346759a8a0559c12df9f9782e05ce2135\": rpc error: code = NotFound desc = could not find container \"36e51f0aeaf3e1d7c32ae217cfd3620346759a8a0559c12df9f9782e05ce2135\": container with ID starting with 36e51f0aeaf3e1d7c32ae217cfd3620346759a8a0559c12df9f9782e05ce2135 not found: ID does not exist" Nov 03 20:08:00.399746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:00.399702 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm"] Nov 03 20:08:00.408434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:00.408397 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-323cb6ed-85f4995966-7v5hm"] Nov 03 20:08:00.637435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:00.637405 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" Nov 03 20:08:01.121123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.121078 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss"] Nov 03 20:08:01.127737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.127697 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-323cb6ed-wztss"] Nov 03 20:08:01.421713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.415903 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt"] Nov 03 20:08:01.421713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.417186 5111 cpu_manager.go:414] "containerMap: removing container" podUID="bb478e55-23a2-4900-884f-c2cabe50a096" containerName="payload-processing" Nov 03 20:08:01.421713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.417210 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb478e55-23a2-4900-884f-c2cabe50a096" containerName="payload-processing" Nov 03 20:08:01.421713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.417292 5111 cpu_manager.go:414] "containerMap: removing container" podUID="cfd0382f-8e90-4ce8-b2de-7d710be227f5" containerName="revoke-keys" Nov 03 20:08:01.421713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.417300 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd0382f-8e90-4ce8-b2de-7d710be227f5" containerName="revoke-keys" Nov 03 20:08:01.421713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.417492 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd0382f-8e90-4ce8-b2de-7d710be227f5" containerName="revoke-keys" Nov 03 20:08:01.421713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.417510 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb478e55-23a2-4900-884f-c2cabe50a096" containerName="payload-processing" Nov 03 20:08:01.478368 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.466471 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.478368 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.478193 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt"] Nov 03 20:08:01.479059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.479034 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-d81d18-openshift-default-dockercfg-7vnl7\"" Nov 03 20:08:01.648746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.648706 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istiod-ca-cert\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.649276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.648779 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-data\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.649276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.648893 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-workload-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.649276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.649026 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-envoy\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.649276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.649045 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-podinfo\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.649276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.649206 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-269w4\" (UniqueName: \"kubernetes.io/projected/4de4cc37-85a3-41b1-b443-b52ccfce94d5-kube-api-access-269w4\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.649276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.649247 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-workload-certs\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.649660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.649352 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-token\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.649660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.649411 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-credential-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.752024 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.751926 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-269w4\" (UniqueName: \"kubernetes.io/projected/4de4cc37-85a3-41b1-b443-b52ccfce94d5-kube-api-access-269w4\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.752024 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.752020 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-workload-certs\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.752277 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.752110 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-token\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.752277 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.752181 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-credential-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.752395 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.752316 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istiod-ca-cert\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.752458 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.752406 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-data\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.752458 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.752438 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-workload-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.752603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.752582 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-envoy\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.752796 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.752772 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-podinfo\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.753802 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.753775 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-credential-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.754586 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.754556 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istiod-ca-cert\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.754966 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.754941 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-workload-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.755060 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.755011 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-data\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.755169 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.755146 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-workload-certs\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.761821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.761753 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-269w4\" (UniqueName: \"kubernetes.io/projected/4de4cc37-85a3-41b1-b443-b52ccfce94d5-kube-api-access-269w4\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.762774 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.762756 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-token\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.772314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.770600 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-podinfo\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.774719 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.774684 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-envoy\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:01.801776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:01.801732 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:02.194848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:02.194810 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f4651e6-ec76-410a-a6fe-a874cf7a5712" path="/var/lib/kubelet/pods/5f4651e6-ec76-410a-a6fe-a874cf7a5712/volumes" Nov 03 20:08:02.196960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:02.196928 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb478e55-23a2-4900-884f-c2cabe50a096" path="/var/lib/kubelet/pods/bb478e55-23a2-4900-884f-c2cabe50a096/volumes" Nov 03 20:08:02.295657 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:08:02.288258 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de4cc37_85a3_41b1_b443_b52ccfce94d5.slice/crio-ddaa194a9ecf38883f4672b9480cb7192a7f702060cb7df8daa27606c9da3b25 WatchSource:0}: Error finding container ddaa194a9ecf38883f4672b9480cb7192a7f702060cb7df8daa27606c9da3b25: Status 404 returned error can't find the container with id ddaa194a9ecf38883f4672b9480cb7192a7f702060cb7df8daa27606c9da3b25 Nov 03 20:08:02.295657 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:02.290807 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt"] Nov 03 20:08:02.295657 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:02.291307 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:08:02.295657 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:02.291375 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:08:02.295657 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:02.291402 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:08:02.537206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:02.535527 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" event={"ID":"4de4cc37-85a3-41b1-b443-b52ccfce94d5","Type":"ContainerStarted","Data":"c71e341f6fc721a00f019a6deecd4c8c5ca238f61d919a369de26cbe7bd0b431"} Nov 03 20:08:02.537206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:02.535614 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" event={"ID":"4de4cc37-85a3-41b1-b443-b52ccfce94d5","Type":"ContainerStarted","Data":"ddaa194a9ecf38883f4672b9480cb7192a7f702060cb7df8daa27606c9da3b25"} Nov 03 20:08:02.561695 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:02.559104 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" podStartSLOduration=1.559087584 podStartE2EDuration="1.559087584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:02.557705204 +0000 UTC m=+1712.520083644" watchObservedRunningTime="2026-11-03 20:08:02.559087584 +0000 UTC m=+1712.521466023" Nov 03 20:08:02.802400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:02.802325 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:03.131557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:03.128783 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c"] Nov 03 20:08:03.361555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:03.353195 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65635a0-wtq9c"] Nov 03 20:08:03.811727 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:03.811682 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 20:08:03.812365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:03.812328 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" podUID="4de4cc37-85a3-41b1-b443-b52ccfce94d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 20:08:04.205121 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:04.205077 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734a8dba-3f4c-4c43-adea-556a98c0d057" path="/var/lib/kubelet/pods/734a8dba-3f4c-4c43-adea-556a98c0d057/volumes" Nov 03 20:08:04.812864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:04.805948 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 20:08:04.812864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:04.806027 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" podUID="4de4cc37-85a3-41b1-b443-b52ccfce94d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 20:08:05.126684 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:08:05.126597 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:08:05.803097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:05.803056 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 20:08:05.803292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:05.803115 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" podUID="4de4cc37-85a3-41b1-b443-b52ccfce94d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 20:08:06.038652 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:06.038616 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t"] Nov 03 20:08:06.088033 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:06.087927 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn"] Nov 03 20:08:06.104302 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:06.103718 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4116667c-vtvbn"] Nov 03 20:08:06.117596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:06.117550 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d01765e-vds4t"] Nov 03 20:08:06.195431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:06.195391 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c82d407-70a8-4f02-a62e-c3c6f0cbd3da" path="/var/lib/kubelet/pods/1c82d407-70a8-4f02-a62e-c3c6f0cbd3da/volumes" Nov 03 20:08:06.196097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:06.196080 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7efe8399-943b-48ac-8555-72198abc6c9d" path="/var/lib/kubelet/pods/7efe8399-943b-48ac-8555-72198abc6c9d/volumes" Nov 03 20:08:06.802640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:06.802600 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 20:08:06.802839 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:06.802661 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" podUID="4de4cc37-85a3-41b1-b443-b52ccfce94d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 20:08:07.028277 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:07.028138 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm"] Nov 03 20:08:07.035382 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:07.035350 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg"] Nov 03 20:08:07.041466 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:07.041431 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad23aea1-bxkgm"] Nov 03 20:08:07.067619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:07.065673 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7340dff2-hpcmg"] Nov 03 20:08:07.802772 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:07.802726 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 20:08:07.802998 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:07.802780 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" podUID="4de4cc37-85a3-41b1-b443-b52ccfce94d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 20:08:08.208287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:08.208236 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680aaedd-5d27-4058-a444-b11825314d65" path="/var/lib/kubelet/pods/680aaedd-5d27-4058-a444-b11825314d65/volumes" Nov 03 20:08:08.210666 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:08.210641 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a347a031-2c7a-46c3-ba61-9825192b7a9e" path="/var/lib/kubelet/pods/a347a031-2c7a-46c3-ba61-9825192b7a9e/volumes" Nov 03 20:08:08.802915 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:08.802878 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 20:08:08.803191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:08.803160 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" podUID="4de4cc37-85a3-41b1-b443-b52ccfce94d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 20:08:09.810513 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:09.810476 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:09.811557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:09.811512 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:09.812314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:09.812290 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:08:11.027249 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:11.027212 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj"] Nov 03 20:08:11.033828 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:11.033790 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5afefd0-lq5wj"] Nov 03 20:08:12.185461 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:12.185425 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d879f60-3aa2-4545-b235-b38eb1b09b96" path="/var/lib/kubelet/pods/6d879f60-3aa2-4545-b235-b38eb1b09b96/volumes" Nov 03 20:08:19.032723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:19.032681 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6"] Nov 03 20:08:19.039332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:19.039297 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2da28f1-2hfc6"] Nov 03 20:08:19.046131 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:19.046104 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7"] Nov 03 20:08:19.051953 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:19.051927 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr"] Nov 03 20:08:19.057190 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:19.057169 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a710ef4-269v7"] Nov 03 20:08:19.062472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:19.062440 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bfd44b2-2khxr"] Nov 03 20:08:20.039824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:20.039791 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4"] Nov 03 20:08:20.046399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:20.046365 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d990d13-6grz4"] Nov 03 20:08:20.244592 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:20.244514 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154e9e64-a6bd-49f4-8b65-6c6ad5541fe4" path="/var/lib/kubelet/pods/154e9e64-a6bd-49f4-8b65-6c6ad5541fe4/volumes" Nov 03 20:08:20.245459 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:20.245435 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2afb37da-7177-4752-93ba-9a68a15a00c8" path="/var/lib/kubelet/pods/2afb37da-7177-4752-93ba-9a68a15a00c8/volumes" Nov 03 20:08:20.246227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:20.246185 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c8f93ca-515a-4785-98e1-1830339ca556" path="/var/lib/kubelet/pods/3c8f93ca-515a-4785-98e1-1830339ca556/volumes" Nov 03 20:08:20.248115 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:20.248095 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f61d390-7b4f-4bc5-a77e-3ea7541f312a" path="/var/lib/kubelet/pods/5f61d390-7b4f-4bc5-a77e-3ea7541f312a/volumes" Nov 03 20:08:26.001426 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:26.001379 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:26.059558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:26.057617 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:26.059558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:26.058003 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:08:26.257614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:26.251753 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:26.257614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:26.255998 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:26.278056 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:26.277822 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:26.316569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:26.316504 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:26.391442 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:26.321136 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:26.690556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:26.689670 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:26.694857 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:26.694812 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:26.733155 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:26.732452 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:26.776678 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:26.776624 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:27.010626 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.009920 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:27.010626 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.010552 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:08:27.021567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.015608 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:27.065556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.062819 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:27.065556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.063212 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 20:08:27.065556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.063785 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:08:27.065556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.063841 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:08:27.267647 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.260623 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:27.326559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.273605 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:27.452571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.295643 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:27.452571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.295683 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:08:27.452571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.308500 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:08:27.452571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.436830 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:08:27.452571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.436876 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:08:27.522619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.518882 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:08:27.522619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.519415 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:08:27.741777 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.741706 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:27.745284 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.745250 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:08:27.783608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.779657 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:27.783608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.779764 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:27.818933 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.818905 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:08:27.843857 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.843815 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:08:27.900036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.887879 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:27.901157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:27.901136 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:08:28.071342 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:28.071233 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:08:28.072042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:28.072001 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:28.072381 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:28.072021 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:08:28.073164 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:28.018100 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:08:28.270664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:28.252909 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:08:28.323378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:28.305804 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:28.418563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:28.416486 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:28.471558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:28.466419 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:28.522555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:28.522351 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:08:28.769554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:28.769151 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:29.060614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.017020 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:08:29.060614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.043251 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:08:29.060614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.050201 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:29.085561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.082481 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:08:29.091584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.090729 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 20:08:29.339476 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.313417 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:08:29.339476 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.337238 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:08:29.422275 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.422241 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:08:29.428693 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.428661 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:08:29.434898 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.433747 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:29.479601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.440766 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:08:29.562648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.543201 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:08:29.617357 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.617282 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:08:29.698928 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.698762 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v"] Nov 03 20:08:29.717719 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.716737 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v"] Nov 03 20:08:29.717719 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.716866 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.727510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.724732 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-0ffd89-openshift-default-dockercfg-rq6r9\"" Nov 03 20:08:29.805040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.804663 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istiod-ca-cert\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.805586 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.805559 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-data\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.805747 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.805729 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-token\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.805897 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.805878 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-workload-certs\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.806016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.806000 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-podinfo\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.806125 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.806110 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-envoy\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.806340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.806301 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-credential-socket\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.806456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.806410 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh5jk\" (UniqueName: \"kubernetes.io/projected/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-kube-api-access-vh5jk\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.806526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.806492 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-workload-socket\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.908557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.908342 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istiod-ca-cert\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.908557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.908400 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-data\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.908557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.908425 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-token\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.908557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.908481 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-workload-certs\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.908557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.908511 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-podinfo\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.908557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.908549 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-envoy\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.909092 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.908612 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-credential-socket\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.909092 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.908644 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh5jk\" (UniqueName: \"kubernetes.io/projected/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-kube-api-access-vh5jk\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.909092 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.908734 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-workload-socket\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.909092 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.908870 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-data\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.909319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.909169 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-credential-socket\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.909319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.909241 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istiod-ca-cert\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.909433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.909352 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-workload-certs\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.909433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.909405 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-workload-socket\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.914501 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.911307 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-podinfo\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.914501 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.911468 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-envoy\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.922089 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.922061 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh5jk\" (UniqueName: \"kubernetes.io/projected/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-kube-api-access-vh5jk\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:29.922501 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:29.922474 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-token\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:30.059554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:30.059006 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:30.601615 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:30.594764 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v"] Nov 03 20:08:30.614310 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:08:30.614234 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1215d5ec_fd6a_40cb_a43c_9a51d68d70f2.slice/crio-9a930ef38a716ad71f0c54a45e45d81fe19971a9f82b0e30afd56c6f18289465 WatchSource:0}: Error finding container 9a930ef38a716ad71f0c54a45e45d81fe19971a9f82b0e30afd56c6f18289465: Status 404 returned error can't find the container with id 9a930ef38a716ad71f0c54a45e45d81fe19971a9f82b0e30afd56c6f18289465 Nov 03 20:08:30.620103 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:30.620066 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:08:30.620279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:30.620263 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:08:30.620365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:30.620354 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:08:31.034591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:31.030055 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27"] Nov 03 20:08:31.060361 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:31.053666 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e00cce-nnv27"] Nov 03 20:08:31.066814 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:31.064183 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr"] Nov 03 20:08:31.073494 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:31.073309 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7578f3b-pzxdr"] Nov 03 20:08:31.676668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:31.676630 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" event={"ID":"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2","Type":"ContainerStarted","Data":"630154985d3ce0bb2176b312942bff177fd8d8a62d52355211ee9750e94bbefe"} Nov 03 20:08:31.677196 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:31.676699 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" event={"ID":"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2","Type":"ContainerStarted","Data":"9a930ef38a716ad71f0c54a45e45d81fe19971a9f82b0e30afd56c6f18289465"} Nov 03 20:08:31.712555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:31.712029 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" podStartSLOduration=3.712009437 podStartE2EDuration="3.712009437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:31.699991488 +0000 UTC m=+1741.662369929" watchObservedRunningTime="2026-11-03 20:08:31.712009437 +0000 UTC m=+1741.674387877" Nov 03 20:08:32.063061 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:32.062963 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:32.069431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:32.069402 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:32.188707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:32.188675 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edca7308-8c50-4292-9b4d-43dbb4334b55" path="/var/lib/kubelet/pods/edca7308-8c50-4292-9b4d-43dbb4334b55/volumes" Nov 03 20:08:32.189226 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:32.189212 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f790e958-814f-41c6-8675-71de6221a2cc" path="/var/lib/kubelet/pods/f790e958-814f-41c6-8675-71de6221a2cc/volumes" Nov 03 20:08:32.690186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:32.690155 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:32.691226 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:32.691204 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:08:33.327203 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.327163 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s"] Nov 03 20:08:33.338227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.334137 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" Nov 03 20:08:33.340713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.340670 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s"] Nov 03 20:08:33.348377 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.348336 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:08:33.353084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.349669 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:08:33.353084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.352828 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:08:33.353084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.352912 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:08:33.453839 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.453802 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865c1010-f54c-4b3d-8dab-68f01d041930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e383a36-f7j6s\" (UID: \"865c1010-f54c-4b3d-8dab-68f01d041930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" Nov 03 20:08:33.454040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.454009 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrdxg\" (UniqueName: \"kubernetes.io/projected/865c1010-f54c-4b3d-8dab-68f01d041930-kube-api-access-vrdxg\") pod \"maas-api-revoke-sub-4e383a36-f7j6s\" (UID: \"865c1010-f54c-4b3d-8dab-68f01d041930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" Nov 03 20:08:33.454117 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.454060 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865c1010-f54c-4b3d-8dab-68f01d041930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e383a36-f7j6s\" (UID: \"865c1010-f54c-4b3d-8dab-68f01d041930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" Nov 03 20:08:33.555827 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.555783 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865c1010-f54c-4b3d-8dab-68f01d041930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e383a36-f7j6s\" (UID: \"865c1010-f54c-4b3d-8dab-68f01d041930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" Nov 03 20:08:33.556021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.555960 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrdxg\" (UniqueName: \"kubernetes.io/projected/865c1010-f54c-4b3d-8dab-68f01d041930-kube-api-access-vrdxg\") pod \"maas-api-revoke-sub-4e383a36-f7j6s\" (UID: \"865c1010-f54c-4b3d-8dab-68f01d041930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" Nov 03 20:08:33.556021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.556003 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865c1010-f54c-4b3d-8dab-68f01d041930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e383a36-f7j6s\" (UID: \"865c1010-f54c-4b3d-8dab-68f01d041930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" Nov 03 20:08:33.556977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.556843 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865c1010-f54c-4b3d-8dab-68f01d041930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e383a36-f7j6s\" (UID: \"865c1010-f54c-4b3d-8dab-68f01d041930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" Nov 03 20:08:33.558583 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.558506 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865c1010-f54c-4b3d-8dab-68f01d041930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e383a36-f7j6s\" (UID: \"865c1010-f54c-4b3d-8dab-68f01d041930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" Nov 03 20:08:33.564340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.564317 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrdxg\" (UniqueName: \"kubernetes.io/projected/865c1010-f54c-4b3d-8dab-68f01d041930-kube-api-access-vrdxg\") pod \"maas-api-revoke-sub-4e383a36-f7j6s\" (UID: \"865c1010-f54c-4b3d-8dab-68f01d041930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" Nov 03 20:08:33.710774 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:33.710739 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" Nov 03 20:08:34.029828 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:34.029798 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s"] Nov 03 20:08:34.033815 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:08:34.033785 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865c1010_f54c_4b3d_8dab_68f01d041930.slice/crio-e3758a5d8bccc86b911a04c17ac1815bddfa47087df54ad01f9e57f1c42c098c WatchSource:0}: Error finding container e3758a5d8bccc86b911a04c17ac1815bddfa47087df54ad01f9e57f1c42c098c: Status 404 returned error can't find the container with id e3758a5d8bccc86b911a04c17ac1815bddfa47087df54ad01f9e57f1c42c098c Nov 03 20:08:34.714898 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:34.714857 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" event={"ID":"865c1010-f54c-4b3d-8dab-68f01d041930","Type":"ContainerStarted","Data":"cc02f0bdf15cc69694990417bb01faf67841276bd2e9e1201e4be71602e62d8f"} Nov 03 20:08:34.715341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:34.714927 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" event={"ID":"865c1010-f54c-4b3d-8dab-68f01d041930","Type":"ContainerStarted","Data":"e3758a5d8bccc86b911a04c17ac1815bddfa47087df54ad01f9e57f1c42c098c"} Nov 03 20:08:34.740827 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:34.740779 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" podStartSLOduration=1.740765573 podStartE2EDuration="1.740765573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:34.733043188 +0000 UTC m=+1744.695421628" watchObservedRunningTime="2026-11-03 20:08:34.740765573 +0000 UTC m=+1744.703144035" Nov 03 20:08:35.095594 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:08:35.088459 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:08:35.161119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:35.161074 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5"] Nov 03 20:08:35.190513 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:35.190477 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5"] Nov 03 20:08:35.190811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:35.190695 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" Nov 03 20:08:35.295754 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:35.294844 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-0ffd89\"" Nov 03 20:08:35.409681 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:35.381356 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtfl6\" (UniqueName: \"kubernetes.io/projected/32f418b9-ac9f-4c9f-bac7-7a3341df7747-kube-api-access-wtfl6\") pod \"payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5\" (UID: \"32f418b9-ac9f-4c9f-bac7-7a3341df7747\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" Nov 03 20:08:35.484661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:35.484622 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32f418b9-ac9f-4c9f-bac7-7a3341df7747-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5\" (UID: \"32f418b9-ac9f-4c9f-bac7-7a3341df7747\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" Nov 03 20:08:35.597555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:35.596556 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtfl6\" (UniqueName: \"kubernetes.io/projected/32f418b9-ac9f-4c9f-bac7-7a3341df7747-kube-api-access-wtfl6\") pod \"payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5\" (UID: \"32f418b9-ac9f-4c9f-bac7-7a3341df7747\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" Nov 03 20:08:35.597555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:35.596721 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32f418b9-ac9f-4c9f-bac7-7a3341df7747-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5\" (UID: \"32f418b9-ac9f-4c9f-bac7-7a3341df7747\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" Nov 03 20:08:35.601777 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:35.601230 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32f418b9-ac9f-4c9f-bac7-7a3341df7747-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5\" (UID: \"32f418b9-ac9f-4c9f-bac7-7a3341df7747\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" Nov 03 20:08:35.612308 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:35.612278 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtfl6\" (UniqueName: \"kubernetes.io/projected/32f418b9-ac9f-4c9f-bac7-7a3341df7747-kube-api-access-wtfl6\") pod \"payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5\" (UID: \"32f418b9-ac9f-4c9f-bac7-7a3341df7747\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" Nov 03 20:08:35.727123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:35.727037 5111 generic.go:356] "Generic (PLEG): container finished" podID="865c1010-f54c-4b3d-8dab-68f01d041930" containerID="cc02f0bdf15cc69694990417bb01faf67841276bd2e9e1201e4be71602e62d8f" exitCode=0 Nov 03 20:08:35.727570 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:35.727142 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" event={"ID":"865c1010-f54c-4b3d-8dab-68f01d041930","Type":"ContainerDied","Data":"cc02f0bdf15cc69694990417bb01faf67841276bd2e9e1201e4be71602e62d8f"} Nov 03 20:08:35.826827 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:35.826787 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" Nov 03 20:08:36.153462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:36.153427 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5"] Nov 03 20:08:36.156694 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:08:36.156666 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f418b9_ac9f_4c9f_bac7_7a3341df7747.slice/crio-039a438d047626174a825391d1da56dfdfc8356a1d6c20f9a0c140803e41a3e9 WatchSource:0}: Error finding container 039a438d047626174a825391d1da56dfdfc8356a1d6c20f9a0c140803e41a3e9: Status 404 returned error can't find the container with id 039a438d047626174a825391d1da56dfdfc8356a1d6c20f9a0c140803e41a3e9 Nov 03 20:08:36.768316 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:36.768264 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" event={"ID":"32f418b9-ac9f-4c9f-bac7-7a3341df7747","Type":"ContainerStarted","Data":"e4a0dba604a5141fefe476e63f1be163fcc025c8c6a79731bfa45863faac54d0"} Nov 03 20:08:36.768921 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:36.768334 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" event={"ID":"32f418b9-ac9f-4c9f-bac7-7a3341df7747","Type":"ContainerStarted","Data":"039a438d047626174a825391d1da56dfdfc8356a1d6c20f9a0c140803e41a3e9"} Nov 03 20:08:36.768921 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:36.768386 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" Nov 03 20:08:36.797012 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:36.796962 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" podStartSLOduration=1.796945616 podStartE2EDuration="1.796945616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:36.785719372 +0000 UTC m=+1746.748097812" watchObservedRunningTime="2026-11-03 20:08:36.796945616 +0000 UTC m=+1746.759324056" Nov 03 20:08:37.104741 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.104712 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" Nov 03 20:08:37.234097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.233195 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/865c1010-f54c-4b3d-8dab-68f01d041930-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865c1010-f54c-4b3d-8dab-68f01d041930-maas-api-xks-ca\") pod \"865c1010-f54c-4b3d-8dab-68f01d041930\" (UID: \"865c1010-f54c-4b3d-8dab-68f01d041930\") " Nov 03 20:08:37.234097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.233411 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865c1010-f54c-4b3d-8dab-68f01d041930-kube-api-access-vrdxg\" (UniqueName: \"kubernetes.io/projected/865c1010-f54c-4b3d-8dab-68f01d041930-kube-api-access-vrdxg\") pod \"865c1010-f54c-4b3d-8dab-68f01d041930\" (UID: \"865c1010-f54c-4b3d-8dab-68f01d041930\") " Nov 03 20:08:37.234097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.233491 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865c1010-f54c-4b3d-8dab-68f01d041930-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865c1010-f54c-4b3d-8dab-68f01d041930-maas-api-openshift-ca\") pod \"865c1010-f54c-4b3d-8dab-68f01d041930\" (UID: \"865c1010-f54c-4b3d-8dab-68f01d041930\") " Nov 03 20:08:37.235658 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.235609 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865c1010-f54c-4b3d-8dab-68f01d041930-maas-api-openshift-ca" pod "865c1010-f54c-4b3d-8dab-68f01d041930" (UID: "865c1010-f54c-4b3d-8dab-68f01d041930"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:37.238068 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.237641 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/865c1010-f54c-4b3d-8dab-68f01d041930-maas-api-xks-ca" pod "865c1010-f54c-4b3d-8dab-68f01d041930" (UID: "865c1010-f54c-4b3d-8dab-68f01d041930"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:37.240981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.240949 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865c1010-f54c-4b3d-8dab-68f01d041930-kube-api-access-vrdxg" pod "865c1010-f54c-4b3d-8dab-68f01d041930" (UID: "865c1010-f54c-4b3d-8dab-68f01d041930"). InnerVolumeSpecName "kube-api-access-vrdxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:37.336226 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.336124 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865c1010-f54c-4b3d-8dab-68f01d041930-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:37.336226 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.336160 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865c1010-f54c-4b3d-8dab-68f01d041930-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:37.336226 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.336174 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrdxg\" (UniqueName: \"kubernetes.io/projected/865c1010-f54c-4b3d-8dab-68f01d041930-kube-api-access-vrdxg\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:37.756417 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.756390 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" Nov 03 20:08:37.756642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.756425 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s" event={"ID":"865c1010-f54c-4b3d-8dab-68f01d041930","Type":"ContainerDied","Data":"e3758a5d8bccc86b911a04c17ac1815bddfa47087df54ad01f9e57f1c42c098c"} Nov 03 20:08:37.756642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.756475 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3758a5d8bccc86b911a04c17ac1815bddfa47087df54ad01f9e57f1c42c098c" Nov 03 20:08:37.758550 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.758512 5111 generic.go:356] "Generic (PLEG): container finished" podID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" containerID="530c601459beaf49d2879234606910984cf2364773161ab50d0305ce6635ecfa" exitCode=0 Nov 03 20:08:37.758678 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.758568 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" event={"ID":"53ba0307-0ce6-43c8-8599-2fb0cd354f61","Type":"ContainerDied","Data":"530c601459beaf49d2879234606910984cf2364773161ab50d0305ce6635ecfa"} Nov 03 20:08:37.759273 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.759254 5111 scope.go:122] "RemoveContainer" containerID="530c601459beaf49d2879234606910984cf2364773161ab50d0305ce6635ecfa" Nov 03 20:08:37.760890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.760861 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:08:37.761021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.760928 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:08:37.761021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:37.760959 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:08:38.795986 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:38.795949 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" event={"ID":"53ba0307-0ce6-43c8-8599-2fb0cd354f61","Type":"ContainerStarted","Data":"bde78d8e8312c48a3f78f696d06ee9bf6e7f13ab37f1f5dc1b5824ed2a6e13e0"} Nov 03 20:08:39.094483 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:39.094394 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:08:39.096461 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:39.096430 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:08:39.096797 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:39.096767 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" podUID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:08:40.078955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:40.078914 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:08:40.079520 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:40.078977 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" podUID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:08:41.081561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:41.081435 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:08:41.081561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:41.081516 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" podUID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:08:42.078864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:42.078823 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:08:42.079096 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:42.078886 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" podUID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:08:43.079619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:43.078866 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:08:43.079619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:43.078940 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" podUID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:08:44.105561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:44.097735 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:08:44.105561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:44.097886 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" podUID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:08:45.126561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:45.123859 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:08:45.126561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:45.125754 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:08:45.139030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:45.138996 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:08:48.799703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:08:48.799665 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" Nov 03 20:09:04.833088 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:04.832660 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:09:05.215901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:05.205654 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:09:05.227232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:05.210507 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:09:05.233965 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:05.212105 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:09:05.270580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:05.213221 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:09:05.270580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:05.213261 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:09:05.270580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:05.216427 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:09:05.318823 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:05.318373 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:09:05.466617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:05.455285 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:09:05.500859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:05.499753 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 20:09:05.500859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:05.500158 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:09:05.582653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:05.582607 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:09:05.665593 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:09:05.661099 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:09:05.845681 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:05.835842 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:09:05.860614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:05.859790 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:09:06.233123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.233077 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:09:06.247013 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.238715 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:09:06.283802 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.238784 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Nov 03 20:09:06.284895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.248321 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:09:06.397722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.397675 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:09:06.441645 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.441600 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:09:06.442576 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.321920 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:09:06.443109 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.321957 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:09:06.444039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.444022 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:09:06.444278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.444263 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:09:06.444522 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.321602 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 20:09:06.453912 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.321870 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:09:06.538821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.513754 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:09:06.673587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.669786 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Nov 03 20:09:06.676261 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.676211 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:09:06.771572 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.760093 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:09:06.801143 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.801055 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:09:06.901717 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.870051 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:09:06.938187 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.938137 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:09:06.938667 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.938646 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:09:06.939004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.938986 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:09:06.939181 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.939166 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:09:06.939419 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.939403 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:09:06.939589 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.939575 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:09:06.960840 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.960817 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:09:06.961210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.961197 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:09:06.961713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:06.961525 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:09:07.010363 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:07.010246 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:09:07.027499 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:07.027469 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:09:07.029847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:07.029731 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" Nov 03 20:09:07.030397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:07.030359 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:09:07.062611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:07.032149 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:09:07.062611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:07.035430 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:09:07.929269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:07.929236 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz"] Nov 03 20:09:07.930084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:07.930064 5111 cpu_manager.go:414] "containerMap: removing container" podUID="865c1010-f54c-4b3d-8dab-68f01d041930" containerName="revoke-keys" Nov 03 20:09:07.930140 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:07.930087 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865c1010-f54c-4b3d-8dab-68f01d041930" containerName="revoke-keys" Nov 03 20:09:07.930245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:07.930236 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="865c1010-f54c-4b3d-8dab-68f01d041930" containerName="revoke-keys" Nov 03 20:09:07.934973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:07.934953 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" Nov 03 20:09:07.938842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:07.938817 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d441dd\"" Nov 03 20:09:07.941802 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:07.941779 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d441dd-dockercfg-4tpc6\"" Nov 03 20:09:07.944235 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:07.944211 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz"] Nov 03 20:09:07.999781 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:07.999751 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdjr5\" (UniqueName: \"kubernetes.io/projected/56cd2469-4b22-4aee-bf4e-a1daf2873729-kube-api-access-gdjr5\") pod \"payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz\" (UID: \"56cd2469-4b22-4aee-bf4e-a1daf2873729\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" Nov 03 20:09:08.000082 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:08.000057 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56cd2469-4b22-4aee-bf4e-a1daf2873729-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz\" (UID: \"56cd2469-4b22-4aee-bf4e-a1daf2873729\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" Nov 03 20:09:08.004295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:08.004263 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp"] Nov 03 20:09:08.019861 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:08.019795 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" Nov 03 20:09:08.037471 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:08.037016 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp"] Nov 03 20:09:08.299612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:08.297405 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdjr5\" (UniqueName: \"kubernetes.io/projected/56cd2469-4b22-4aee-bf4e-a1daf2873729-kube-api-access-gdjr5\") pod \"payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz\" (UID: \"56cd2469-4b22-4aee-bf4e-a1daf2873729\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" Nov 03 20:09:08.299612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:08.297585 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56cd2469-4b22-4aee-bf4e-a1daf2873729-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz\" (UID: \"56cd2469-4b22-4aee-bf4e-a1daf2873729\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" Nov 03 20:09:08.434623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:08.407707 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56cd2469-4b22-4aee-bf4e-a1daf2873729-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz\" (UID: \"56cd2469-4b22-4aee-bf4e-a1daf2873729\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" Nov 03 20:09:08.437655 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:08.437568 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdjr5\" (UniqueName: \"kubernetes.io/projected/56cd2469-4b22-4aee-bf4e-a1daf2873729-kube-api-access-gdjr5\") pod \"payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz\" (UID: \"56cd2469-4b22-4aee-bf4e-a1daf2873729\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" Nov 03 20:09:08.508140 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:08.508098 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4cc53c8-f835-4d6c-aa61-7369d62ae1db-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp\" (UID: \"c4cc53c8-f835-4d6c-aa61-7369d62ae1db\") " pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" Nov 03 20:09:08.508317 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:08.508178 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td54x\" (UniqueName: \"kubernetes.io/projected/c4cc53c8-f835-4d6c-aa61-7369d62ae1db-kube-api-access-td54x\") pod \"payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp\" (UID: \"c4cc53c8-f835-4d6c-aa61-7369d62ae1db\") " pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" Nov 03 20:09:08.596782 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:08.595935 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" Nov 03 20:09:08.609389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:08.609359 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4cc53c8-f835-4d6c-aa61-7369d62ae1db-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp\" (UID: \"c4cc53c8-f835-4d6c-aa61-7369d62ae1db\") " pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" Nov 03 20:09:08.609627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:08.609420 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td54x\" (UniqueName: \"kubernetes.io/projected/c4cc53c8-f835-4d6c-aa61-7369d62ae1db-kube-api-access-td54x\") pod \"payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp\" (UID: \"c4cc53c8-f835-4d6c-aa61-7369d62ae1db\") " pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" Nov 03 20:09:08.611462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:08.610842 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4cc53c8-f835-4d6c-aa61-7369d62ae1db-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp\" (UID: \"c4cc53c8-f835-4d6c-aa61-7369d62ae1db\") " pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" Nov 03 20:09:08.618320 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:08.618294 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td54x\" (UniqueName: \"kubernetes.io/projected/c4cc53c8-f835-4d6c-aa61-7369d62ae1db-kube-api-access-td54x\") pod \"payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp\" (UID: \"c4cc53c8-f835-4d6c-aa61-7369d62ae1db\") " pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" Nov 03 20:09:08.778655 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:08.778621 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" Nov 03 20:09:09.028448 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:09.012706 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz"] Nov 03 20:09:09.193460 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:09:09.193423 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4cc53c8_f835_4d6c_aa61_7369d62ae1db.slice/crio-827206bae471c0a78d19008b849fcd35e6d5467dc1cff87c0d67a80167a16014 WatchSource:0}: Error finding container 827206bae471c0a78d19008b849fcd35e6d5467dc1cff87c0d67a80167a16014: Status 404 returned error can't find the container with id 827206bae471c0a78d19008b849fcd35e6d5467dc1cff87c0d67a80167a16014 Nov 03 20:09:09.195774 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:09.195749 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp"] Nov 03 20:09:09.706250 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:09.706208 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" event={"ID":"c4cc53c8-f835-4d6c-aa61-7369d62ae1db","Type":"ContainerStarted","Data":"09fa221b587eb8aa31b28a672fab43a9e44c979d970814dcd6b4180be677f5fb"} Nov 03 20:09:09.706466 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:09.706282 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" event={"ID":"c4cc53c8-f835-4d6c-aa61-7369d62ae1db","Type":"ContainerStarted","Data":"827206bae471c0a78d19008b849fcd35e6d5467dc1cff87c0d67a80167a16014"} Nov 03 20:09:09.706466 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:09.706315 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" Nov 03 20:09:09.708061 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:09.708034 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" event={"ID":"56cd2469-4b22-4aee-bf4e-a1daf2873729","Type":"ContainerStarted","Data":"c84d9bb6f0a029df16cb5679ed1d6f7e77b9d011e492b5cb52d55a239e8b85ec"} Nov 03 20:09:09.708184 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:09.708083 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" event={"ID":"56cd2469-4b22-4aee-bf4e-a1daf2873729","Type":"ContainerStarted","Data":"74812fca06eb3272ee0c610793e2e38684efaed24da4f8e75caa19bcfe167daf"} Nov 03 20:09:09.708184 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:09.708109 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" Nov 03 20:09:09.730040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:09.729989 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" podStartSLOduration=2.729970836 podStartE2EDuration="2.729970836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:09.722320211 +0000 UTC m=+1779.684698666" watchObservedRunningTime="2026-11-03 20:09:09.729970836 +0000 UTC m=+1779.692349275" Nov 03 20:09:09.741723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:09.741681 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" podStartSLOduration=2.741670193 podStartE2EDuration="2.741670193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:09.738876397 +0000 UTC m=+1779.701254837" watchObservedRunningTime="2026-11-03 20:09:09.741670193 +0000 UTC m=+1779.704048677" Nov 03 20:09:15.782955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:15.782916 5111 generic.go:356] "Generic (PLEG): container finished" podID="1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" containerID="630154985d3ce0bb2176b312942bff177fd8d8a62d52355211ee9750e94bbefe" exitCode=0 Nov 03 20:09:15.783470 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:15.782987 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" event={"ID":"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2","Type":"ContainerDied","Data":"630154985d3ce0bb2176b312942bff177fd8d8a62d52355211ee9750e94bbefe"} Nov 03 20:09:15.783586 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:15.783523 5111 scope.go:122] "RemoveContainer" containerID="630154985d3ce0bb2176b312942bff177fd8d8a62d52355211ee9750e94bbefe" Nov 03 20:09:15.785452 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:15.785418 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:09:15.785602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:15.785508 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:09:15.785602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:15.785594 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:09:16.795955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:16.795916 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" event={"ID":"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2","Type":"ContainerStarted","Data":"54d095a0b7b62c3ae44ba288a768ae9a5ec315a2df862692323fa8e00c2a468e"} Nov 03 20:09:17.088937 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:17.087931 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:09:17.090857 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:17.090584 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:09:17.093824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:17.093782 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" podUID="1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 20:09:17.691580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:17.690809 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:09:18.071609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:18.066631 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:09:18.071609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:18.066695 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" podUID="1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 20:09:19.071967 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:19.071930 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:09:19.072405 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:19.071990 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" podUID="1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 20:09:20.063556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:20.063005 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:09:20.063556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:20.063085 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" podUID="1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 20:09:20.723638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:20.723606 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" Nov 03 20:09:20.724080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:20.723690 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" Nov 03 20:09:21.083056 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:21.082962 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:09:21.085808 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:21.084384 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" podUID="1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 20:09:22.063547 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:22.063501 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:09:22.063970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:22.063581 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" podUID="1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 20:09:23.071045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:23.071010 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:09:23.073232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:23.073202 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:09:25.736506 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.736467 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d"] Nov 03 20:09:25.745501 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.743342 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.754636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.754607 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1962f9-openshift-default-dockercfg-qbhsx\"" Nov 03 20:09:25.769035 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.768998 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d"] Nov 03 20:09:25.852859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.850123 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-credential-socket\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.852859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.850246 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-podinfo\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.852859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.850272 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17938ed6-2f40-4c8a-84f4-97bda29142cc-istiod-ca-cert\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.852859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.850379 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-envoy\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.852859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.850406 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-token\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.852859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.850566 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-workload-certs\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.852859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.850751 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-workload-socket\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.852859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.850923 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-data\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.852859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.850956 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5vvs\" (UniqueName: \"kubernetes.io/projected/17938ed6-2f40-4c8a-84f4-97bda29142cc-kube-api-access-n5vvs\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.963556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.955695 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-workload-certs\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.963556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.955872 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-workload-socket\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.963556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.956007 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-data\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.963556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.956035 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5vvs\" (UniqueName: \"kubernetes.io/projected/17938ed6-2f40-4c8a-84f4-97bda29142cc-kube-api-access-n5vvs\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.963556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.956087 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-credential-socket\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.963556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.956155 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-podinfo\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.963556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.956180 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17938ed6-2f40-4c8a-84f4-97bda29142cc-istiod-ca-cert\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.963556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.956263 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-envoy\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.963556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.956286 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-token\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.963556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.957142 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-workload-certs\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.963556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.957378 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-workload-socket\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.963556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.957667 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-data\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.963556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.958211 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-credential-socket\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:25.963556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.961272 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17938ed6-2f40-4c8a-84f4-97bda29142cc-istiod-ca-cert\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:26.016605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:25.998056 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-podinfo\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:26.016842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:26.016765 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5vvs\" (UniqueName: \"kubernetes.io/projected/17938ed6-2f40-4c8a-84f4-97bda29142cc-kube-api-access-n5vvs\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:26.021579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:26.017488 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-envoy\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:26.088509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:26.088459 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-token\") pod \"e2e-api-a-1962f9-openshift-default-56464595f-d2j7d\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:26.378485 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:26.378441 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:26.688980 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:09:26.688948 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17938ed6_2f40_4c8a_84f4_97bda29142cc.slice/crio-f2e3dd3737b4da16e98db05a1943bd08e69b490308516e017f3516dda537bef1 WatchSource:0}: Error finding container f2e3dd3737b4da16e98db05a1943bd08e69b490308516e017f3516dda537bef1: Status 404 returned error can't find the container with id f2e3dd3737b4da16e98db05a1943bd08e69b490308516e017f3516dda537bef1 Nov 03 20:09:26.691867 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:26.691840 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d"] Nov 03 20:09:26.692577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:26.692545 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:09:26.692633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:26.692622 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:09:26.692672 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:26.692651 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:09:27.105581 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:27.105464 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" event={"ID":"17938ed6-2f40-4c8a-84f4-97bda29142cc","Type":"ContainerStarted","Data":"aa025bd892cda3c538628a7c854d8a16a862172523c2d43a2b1ab3209f664e23"} Nov 03 20:09:27.105974 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:27.105618 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" event={"ID":"17938ed6-2f40-4c8a-84f4-97bda29142cc","Type":"ContainerStarted","Data":"f2e3dd3737b4da16e98db05a1943bd08e69b490308516e017f3516dda537bef1"} Nov 03 20:09:27.142334 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:27.142264 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" podStartSLOduration=2.142244326 podStartE2EDuration="2.142244326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:27.130384159 +0000 UTC m=+1797.092762623" watchObservedRunningTime="2026-11-03 20:09:27.142244326 +0000 UTC m=+1797.104622767" Nov 03 20:09:27.379110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:27.379072 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:27.386460 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:27.386410 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:28.115990 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:28.115954 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:28.117007 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:28.116985 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:09:30.551254 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.551205 5111 scope.go:122] "RemoveContainer" containerID="0952d82be61664a23bcecd5bd289bd2be61ce5cbacfcc596b1968be9ee794dd0" Nov 03 20:09:30.572245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.572216 5111 scope.go:122] "RemoveContainer" containerID="50dec5ac6600d2651005dfb9e8db5e692901ad5b364c530e302b80a41f76c7d9" Nov 03 20:09:30.588519 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.588456 5111 scope.go:122] "RemoveContainer" containerID="9cc7269e0c250bb3221fa1355a13f61ef7fc07abfdc3f33b03752b30e1f30c6a" Nov 03 20:09:30.592468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.592442 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:09:30.599747 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.599691 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:09:30.602016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.594688 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:09:30.605642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.605623 5111 scope.go:122] "RemoveContainer" containerID="5c162edccce3381f64f00187b93347949b59e847ed155d54b16eabc79c03a626" Nov 03 20:09:30.611619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.611601 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:09:30.624357 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.624337 5111 scope.go:122] "RemoveContainer" containerID="6c5b084cf045679bcd8c9bb4280b8b951408ddd6f94e3c62c5550d380f1c7c53" Nov 03 20:09:30.636391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.636369 5111 scope.go:122] "RemoveContainer" containerID="06d6fa60fa9803a16aae4956a95a6d7ee1c6e1251376bc3b1c06a0c46f191e21" Nov 03 20:09:30.647042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.647024 5111 scope.go:122] "RemoveContainer" containerID="b86c97eaeee25478dec82ee77cbc6a6335702c97c722ba45205c24aad27187fb" Nov 03 20:09:30.657322 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.657293 5111 scope.go:122] "RemoveContainer" containerID="7ed457a3a1c41c29543fd8ec45d98fa48d60761c13539ffc477fc4444dc91adb" Nov 03 20:09:30.668217 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.668196 5111 scope.go:122] "RemoveContainer" containerID="1e793ad93b7e6ee16a65000aaeae957d61caa31cf9da8a30f70d9d9ccab59505" Nov 03 20:09:30.678287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.678269 5111 scope.go:122] "RemoveContainer" containerID="e2b41cf3186dfd4e278fc366a37e5bc83ff9387d52435ff1827a542ad381804e" Nov 03 20:09:30.689402 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.689378 5111 scope.go:122] "RemoveContainer" containerID="11adb005aaf0cb005283d94794ee746ac76b490b77f680eb8462e851c3509ec3" Nov 03 20:09:30.700372 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.700351 5111 scope.go:122] "RemoveContainer" containerID="d22be5826698be97336a7022df2962d34d3f7150ef64d8935895d1903c0f1b6b" Nov 03 20:09:30.710357 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.710340 5111 scope.go:122] "RemoveContainer" containerID="582fe2e7839287c561417bc6f4849b9cdba041c406760c819b5a7a89b431071d" Nov 03 20:09:30.723083 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.723056 5111 scope.go:122] "RemoveContainer" containerID="463b5e920fd3d14306f88d47dbd587a50e5f34f8c538fb01415d5c1c8fb9a8ad" Nov 03 20:09:30.733936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.733904 5111 scope.go:122] "RemoveContainer" containerID="4da38a879655be99d50e7f4d7c779ed486539dacfb84e8567961b0bef8c40a08" Nov 03 20:09:30.744565 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.744545 5111 scope.go:122] "RemoveContainer" containerID="94940369510c9332d811c3a67cdf72a00cc696fdeae10b1bafba2b1fd6e27fa7" Nov 03 20:09:30.755552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.755506 5111 scope.go:122] "RemoveContainer" containerID="239bc6379a457163e4a618c4a5c5f881ef622a8efc04f9e9e0b0ec8dd0a75a93" Nov 03 20:09:30.953878 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.953841 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7"] Nov 03 20:09:30.963522 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.963496 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" Nov 03 20:09:30.971255 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.971229 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7"] Nov 03 20:09:30.978642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:30.978617 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1962f9\"" Nov 03 20:09:31.000352 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:31.000265 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2064f1d-c7ba-4dd0-9dbe-e71e4a120956-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7\" (UID: \"f2064f1d-c7ba-4dd0-9dbe-e71e4a120956\") " pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" Nov 03 20:09:31.000714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:31.000383 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhmqv\" (UniqueName: \"kubernetes.io/projected/f2064f1d-c7ba-4dd0-9dbe-e71e4a120956-kube-api-access-vhmqv\") pod \"payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7\" (UID: \"f2064f1d-c7ba-4dd0-9dbe-e71e4a120956\") " pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" Nov 03 20:09:31.130617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:31.109918 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2064f1d-c7ba-4dd0-9dbe-e71e4a120956-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7\" (UID: \"f2064f1d-c7ba-4dd0-9dbe-e71e4a120956\") " pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" Nov 03 20:09:31.130949 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:31.130915 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhmqv\" (UniqueName: \"kubernetes.io/projected/f2064f1d-c7ba-4dd0-9dbe-e71e4a120956-kube-api-access-vhmqv\") pod \"payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7\" (UID: \"f2064f1d-c7ba-4dd0-9dbe-e71e4a120956\") " pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" Nov 03 20:09:31.153125 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:31.153090 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhmqv\" (UniqueName: \"kubernetes.io/projected/f2064f1d-c7ba-4dd0-9dbe-e71e4a120956-kube-api-access-vhmqv\") pod \"payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7\" (UID: \"f2064f1d-c7ba-4dd0-9dbe-e71e4a120956\") " pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" Nov 03 20:09:31.171562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:31.167062 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2064f1d-c7ba-4dd0-9dbe-e71e4a120956-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7\" (UID: \"f2064f1d-c7ba-4dd0-9dbe-e71e4a120956\") " pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" Nov 03 20:09:31.319886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:31.309671 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" Nov 03 20:09:31.728485 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:31.728452 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7"] Nov 03 20:09:32.339398 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:32.339352 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" event={"ID":"f2064f1d-c7ba-4dd0-9dbe-e71e4a120956","Type":"ContainerStarted","Data":"dca93f6712d3b50fc96fa4928b03f37923abf68a2f02a5dd216ef3cd72dabdc9"} Nov 03 20:09:32.339614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:32.339414 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" event={"ID":"f2064f1d-c7ba-4dd0-9dbe-e71e4a120956","Type":"ContainerStarted","Data":"4644686cd82cff97bd3fac3cccdeff9212f4ac7f6f37abecefc8dd9803d92a5e"} Nov 03 20:09:32.339683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:32.339604 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" Nov 03 20:09:32.358932 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:32.358881 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" podStartSLOduration=2.358867956 podStartE2EDuration="2.358867956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:32.355103917 +0000 UTC m=+1802.317482354" watchObservedRunningTime="2026-11-03 20:09:32.358867956 +0000 UTC m=+1802.321246402" Nov 03 20:09:33.232614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.232575 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p"] Nov 03 20:09:33.239228 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.239201 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.248583 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.248231 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p"] Nov 03 20:09:33.253193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.252758 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-cad531-openshift-default-dockercfg-gj4fh\"" Nov 03 20:09:33.343668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.329910 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.343668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.330046 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.343668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.330066 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.343668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.330097 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6djp\" (UniqueName: \"kubernetes.io/projected/7de3942d-307c-4f4c-b291-2b69ad18f472-kube-api-access-t6djp\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.343668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.330168 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.343668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.330212 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.343668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.330251 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.343668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.330312 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.438747 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.438706 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de3942d-307c-4f4c-b291-2b69ad18f472-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.563634 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.542908 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de3942d-307c-4f4c-b291-2b69ad18f472-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.581551 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.580393 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de3942d-307c-4f4c-b291-2b69ad18f472-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.582601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.582528 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.583298 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.583278 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.586774 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.586755 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.587333 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.587310 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6djp\" (UniqueName: \"kubernetes.io/projected/7de3942d-307c-4f4c-b291-2b69ad18f472-kube-api-access-t6djp\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.588819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.588796 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.589413 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.589393 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.589866 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.589848 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.590469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.590446 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.591082 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.589292 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.591263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.583126 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.591403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.590858 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.591521 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.589787 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.644423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.644390 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6djp\" (UniqueName: \"kubernetes.io/projected/7de3942d-307c-4f4c-b291-2b69ad18f472-kube-api-access-t6djp\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.660641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.645133 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.660944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.659239 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.691303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.691264 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:33.867027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:33.866935 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:34.188734 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:09:34.188696 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de3942d_307c_4f4c_b291_2b69ad18f472.slice/crio-2f355898d64850fc4dabec5dfddd21846af1eb0e30a5785290d2dbd1a5a7c643 WatchSource:0}: Error finding container 2f355898d64850fc4dabec5dfddd21846af1eb0e30a5785290d2dbd1a5a7c643: Status 404 returned error can't find the container with id 2f355898d64850fc4dabec5dfddd21846af1eb0e30a5785290d2dbd1a5a7c643 Nov 03 20:09:34.189695 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:34.189668 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p"] Nov 03 20:09:34.192052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:34.192015 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:09:34.192144 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:34.192097 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:09:34.192212 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:34.192145 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:09:34.482677 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:34.482583 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" event={"ID":"7de3942d-307c-4f4c-b291-2b69ad18f472","Type":"ContainerStarted","Data":"1e8d76fd89609448f8f6a2d4f962288558792de0ccea2dc41e0e394942b9f617"} Nov 03 20:09:34.482677 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:34.482663 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" event={"ID":"7de3942d-307c-4f4c-b291-2b69ad18f472","Type":"ContainerStarted","Data":"2f355898d64850fc4dabec5dfddd21846af1eb0e30a5785290d2dbd1a5a7c643"} Nov 03 20:09:34.505828 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:34.505777 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" podStartSLOduration=1.505763003 podStartE2EDuration="1.505763003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:34.502990827 +0000 UTC m=+1804.465369266" watchObservedRunningTime="2026-11-03 20:09:34.505763003 +0000 UTC m=+1804.468141441" Nov 03 20:09:34.867160 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:34.867081 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:35.072004 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:09:35.071960 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:09:35.874210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:35.874176 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:36.555851 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:36.555816 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:36.558193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:36.558166 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:09:37.055653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:37.055614 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz"] Nov 03 20:09:37.060449 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:37.060420 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2514ae91-28pwz"] Nov 03 20:09:38.194309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.194277 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb95e881-d3b3-4b48-9242-c05140b94826" path="/var/lib/kubelet/pods/cb95e881-d3b3-4b48-9242-c05140b94826/volumes" Nov 03 20:09:38.289644 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.289611 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb"] Nov 03 20:09:38.295835 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.295810 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" Nov 03 20:09:38.313423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.313398 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-cad531\"" Nov 03 20:09:38.314365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.314340 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-cad531-dockercfg-kz6dt\"" Nov 03 20:09:38.336225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.336193 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb"] Nov 03 20:09:38.428563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.399268 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbdfc\" (UniqueName: \"kubernetes.io/projected/407c2001-b858-4378-a649-c300979e4f85-kube-api-access-bbdfc\") pod \"payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb\" (UID: \"407c2001-b858-4378-a649-c300979e4f85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" Nov 03 20:09:38.462078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.461980 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407c2001-b858-4378-a649-c300979e4f85-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb\" (UID: \"407c2001-b858-4378-a649-c300979e4f85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" Nov 03 20:09:38.600312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.600277 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407c2001-b858-4378-a649-c300979e4f85-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb\" (UID: \"407c2001-b858-4378-a649-c300979e4f85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" Nov 03 20:09:38.618723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.618210 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbdfc\" (UniqueName: \"kubernetes.io/projected/407c2001-b858-4378-a649-c300979e4f85-kube-api-access-bbdfc\") pod \"payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb\" (UID: \"407c2001-b858-4378-a649-c300979e4f85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" Nov 03 20:09:38.621905 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.620179 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407c2001-b858-4378-a649-c300979e4f85-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb\" (UID: \"407c2001-b858-4378-a649-c300979e4f85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" Nov 03 20:09:38.657883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.652389 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p"] Nov 03 20:09:38.679325 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.662704 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" Nov 03 20:09:38.720717 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.720612 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58qvs\" (UniqueName: \"kubernetes.io/projected/2aa57a2e-9b53-4c9e-a17c-48f12faaac69-kube-api-access-58qvs\") pod \"payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p\" (UID: \"2aa57a2e-9b53-4c9e-a17c-48f12faaac69\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" Nov 03 20:09:38.720956 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.720920 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa57a2e-9b53-4c9e-a17c-48f12faaac69-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p\" (UID: \"2aa57a2e-9b53-4c9e-a17c-48f12faaac69\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" Nov 03 20:09:38.723674 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.723648 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbdfc\" (UniqueName: \"kubernetes.io/projected/407c2001-b858-4378-a649-c300979e4f85-kube-api-access-bbdfc\") pod \"payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb\" (UID: \"407c2001-b858-4378-a649-c300979e4f85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" Nov 03 20:09:38.738466 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.738424 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p"] Nov 03 20:09:38.832373 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.832326 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58qvs\" (UniqueName: \"kubernetes.io/projected/2aa57a2e-9b53-4c9e-a17c-48f12faaac69-kube-api-access-58qvs\") pod \"payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p\" (UID: \"2aa57a2e-9b53-4c9e-a17c-48f12faaac69\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" Nov 03 20:09:38.832692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.832657 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa57a2e-9b53-4c9e-a17c-48f12faaac69-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p\" (UID: \"2aa57a2e-9b53-4c9e-a17c-48f12faaac69\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" Nov 03 20:09:38.834752 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.834723 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa57a2e-9b53-4c9e-a17c-48f12faaac69-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p\" (UID: \"2aa57a2e-9b53-4c9e-a17c-48f12faaac69\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" Nov 03 20:09:38.841486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.841458 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58qvs\" (UniqueName: \"kubernetes.io/projected/2aa57a2e-9b53-4c9e-a17c-48f12faaac69-kube-api-access-58qvs\") pod \"payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p\" (UID: \"2aa57a2e-9b53-4c9e-a17c-48f12faaac69\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" Nov 03 20:09:38.918855 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:38.918812 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" Nov 03 20:09:39.041564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.037773 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" Nov 03 20:09:39.286362 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:09:39.286328 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407c2001_b858_4378_a649_c300979e4f85.slice/crio-ffa2fa11a0537401821e84ea7be80be38622be4ae4fc1bbcb3a3681784856795 WatchSource:0}: Error finding container ffa2fa11a0537401821e84ea7be80be38622be4ae4fc1bbcb3a3681784856795: Status 404 returned error can't find the container with id ffa2fa11a0537401821e84ea7be80be38622be4ae4fc1bbcb3a3681784856795 Nov 03 20:09:39.289809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.289779 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb"] Nov 03 20:09:39.401026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.400995 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd"] Nov 03 20:09:39.407295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.407271 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" Nov 03 20:09:39.420024 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.419990 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b9dc48\"" Nov 03 20:09:39.424801 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.423269 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd"] Nov 03 20:09:39.576606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.521838 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b9dc48-dockercfg-42kxp\"" Nov 03 20:09:39.595873 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.595667 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeab908d-cb84-4a21-8caf-7b7434c3b0e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd\" (UID: \"aeab908d-cb84-4a21-8caf-7b7434c3b0e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" Nov 03 20:09:39.595873 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.595746 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2zqt\" (UniqueName: \"kubernetes.io/projected/aeab908d-cb84-4a21-8caf-7b7434c3b0e8-kube-api-access-g2zqt\") pod \"payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd\" (UID: \"aeab908d-cb84-4a21-8caf-7b7434c3b0e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" Nov 03 20:09:39.716557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.686209 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p"] Nov 03 20:09:39.716557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.712294 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeab908d-cb84-4a21-8caf-7b7434c3b0e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd\" (UID: \"aeab908d-cb84-4a21-8caf-7b7434c3b0e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" Nov 03 20:09:39.716557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.712345 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2zqt\" (UniqueName: \"kubernetes.io/projected/aeab908d-cb84-4a21-8caf-7b7434c3b0e8-kube-api-access-g2zqt\") pod \"payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd\" (UID: \"aeab908d-cb84-4a21-8caf-7b7434c3b0e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" Nov 03 20:09:39.792101 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.731991 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeab908d-cb84-4a21-8caf-7b7434c3b0e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd\" (UID: \"aeab908d-cb84-4a21-8caf-7b7434c3b0e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" Nov 03 20:09:39.792376 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.792340 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2zqt\" (UniqueName: \"kubernetes.io/projected/aeab908d-cb84-4a21-8caf-7b7434c3b0e8-kube-api-access-g2zqt\") pod \"payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd\" (UID: \"aeab908d-cb84-4a21-8caf-7b7434c3b0e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" Nov 03 20:09:39.863813 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.863766 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" event={"ID":"407c2001-b858-4378-a649-c300979e4f85","Type":"ContainerStarted","Data":"9e0b64c0a06dfefd53aa759f7b81202409a6ef646931d4072191e8fda51503c3"} Nov 03 20:09:39.864008 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.863843 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" event={"ID":"407c2001-b858-4378-a649-c300979e4f85","Type":"ContainerStarted","Data":"ffa2fa11a0537401821e84ea7be80be38622be4ae4fc1bbcb3a3681784856795"} Nov 03 20:09:39.865404 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.865378 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" Nov 03 20:09:39.869579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.868287 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" event={"ID":"2aa57a2e-9b53-4c9e-a17c-48f12faaac69","Type":"ContainerStarted","Data":"e94428cb0e3b09918bf4c728c505228c57ab84a6da3b4ace11b481eddce48090"} Nov 03 20:09:39.888470 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:39.888415 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" podStartSLOduration=1.888398488 podStartE2EDuration="1.888398488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:39.882583343 +0000 UTC m=+1809.844961787" watchObservedRunningTime="2026-11-03 20:09:39.888398488 +0000 UTC m=+1809.850776974" Nov 03 20:09:40.045193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:40.045152 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" Nov 03 20:09:40.445762 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:09:40.445727 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeab908d_cb84_4a21_8caf_7b7434c3b0e8.slice/crio-1e5ef135d569ed8cd0ac1fe1bdbc1f49d0eeab74d978aa49ccd1a7095e770c49 WatchSource:0}: Error finding container 1e5ef135d569ed8cd0ac1fe1bdbc1f49d0eeab74d978aa49ccd1a7095e770c49: Status 404 returned error can't find the container with id 1e5ef135d569ed8cd0ac1fe1bdbc1f49d0eeab74d978aa49ccd1a7095e770c49 Nov 03 20:09:40.447308 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:40.447283 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd"] Nov 03 20:09:40.880496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:40.880460 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" event={"ID":"aeab908d-cb84-4a21-8caf-7b7434c3b0e8","Type":"ContainerStarted","Data":"02624675b1794c5ab4351316170d302442e36efbce63b1d3bb01a7b8b4bc6267"} Nov 03 20:09:40.880732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:40.880516 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" event={"ID":"aeab908d-cb84-4a21-8caf-7b7434c3b0e8","Type":"ContainerStarted","Data":"1e5ef135d569ed8cd0ac1fe1bdbc1f49d0eeab74d978aa49ccd1a7095e770c49"} Nov 03 20:09:40.880732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:40.880593 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" Nov 03 20:09:40.882625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:40.882581 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" event={"ID":"2aa57a2e-9b53-4c9e-a17c-48f12faaac69","Type":"ContainerStarted","Data":"f1d3f087ba08d53554d5d78716981a646d6b0bd78515a3fa1f404feef6c7190b"} Nov 03 20:09:40.882770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:40.882648 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" Nov 03 20:09:40.900600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:40.900525 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" podStartSLOduration=1.9005084650000001 podStartE2EDuration="1.900508465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:40.898031369 +0000 UTC m=+1810.860409808" watchObservedRunningTime="2026-11-03 20:09:40.900508465 +0000 UTC m=+1810.862886906" Nov 03 20:09:40.923194 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:40.923131 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" podStartSLOduration=2.923114213 podStartE2EDuration="2.923114213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:40.915762219 +0000 UTC m=+1810.878140659" watchObservedRunningTime="2026-11-03 20:09:40.923114213 +0000 UTC m=+1810.885492652" Nov 03 20:09:43.471284 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:43.471252 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" Nov 03 20:09:51.900097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:51.900056 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" Nov 03 20:09:51.903636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:51.901984 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" Nov 03 20:09:51.903968 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:51.903944 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" Nov 03 20:09:59.027641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.027606 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm"] Nov 03 20:09:59.038559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.035369 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9"] Nov 03 20:09:59.044615 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.044586 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c86072-j86mm"] Nov 03 20:09:59.053106 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.053076 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5d1e66-xvvs9"] Nov 03 20:09:59.404553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.403897 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww"] Nov 03 20:09:59.418317 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.412286 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.425638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.424259 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-acd98d21-openshift-default-dockercfg-b8v8z\"" Nov 03 20:09:59.441297 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.441272 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww"] Nov 03 20:09:59.603286 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.603247 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istiod-ca-cert\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.603476 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.603313 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-workload-certs\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.603476 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.603397 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-credential-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.603577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.603510 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-envoy\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.603620 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.603594 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-podinfo\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.603664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.603651 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-token\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.603811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.603799 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-data\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.603847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.603829 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-workload-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.603909 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.603894 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqqtv\" (UniqueName: \"kubernetes.io/projected/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-kube-api-access-pqqtv\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.705364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.705277 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-podinfo\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.705364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.705334 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-token\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.705683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.705510 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-data\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.705683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.705596 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-workload-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.705683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.705661 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqqtv\" (UniqueName: \"kubernetes.io/projected/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-kube-api-access-pqqtv\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.705851 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.705772 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istiod-ca-cert\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.705851 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.705837 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-workload-certs\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.705961 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.705879 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-credential-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.705961 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.705895 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-data\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.705961 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.705936 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-workload-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.706132 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.706019 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-envoy\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.706244 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.706224 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-credential-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.706316 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.706287 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-workload-certs\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.706743 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.706716 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istiod-ca-cert\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.708016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.707991 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-podinfo\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.708179 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.708161 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-envoy\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.712942 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.712919 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-token\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.713658 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.713639 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqqtv\" (UniqueName: \"kubernetes.io/projected/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-kube-api-access-pqqtv\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:09:59.750230 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:09:59.750203 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:10:00.153345 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:10:00.153299 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3172c55_00d5_4500_b74e_e0d0cf88cb2e.slice/crio-9fa6376f3ba083f60f25e75401533db4602619bc61d4d79e66bb2b6005d0ff7f WatchSource:0}: Error finding container 9fa6376f3ba083f60f25e75401533db4602619bc61d4d79e66bb2b6005d0ff7f: Status 404 returned error can't find the container with id 9fa6376f3ba083f60f25e75401533db4602619bc61d4d79e66bb2b6005d0ff7f Nov 03 20:10:00.154631 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:00.154309 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww"] Nov 03 20:10:00.165275 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:00.165239 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:00.165439 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:00.165330 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:00.165439 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:00.165380 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:00.195375 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:00.195341 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1af6db2e-d6c9-47fd-a41e-c412988701d7" path="/var/lib/kubelet/pods/1af6db2e-d6c9-47fd-a41e-c412988701d7/volumes" Nov 03 20:10:00.196480 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:00.196448 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e478b9aa-0913-49f0-82de-15d52734181f" path="/var/lib/kubelet/pods/e478b9aa-0913-49f0-82de-15d52734181f/volumes" Nov 03 20:10:00.199310 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:00.199265 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" event={"ID":"a3172c55-00d5-4500-b74e-e0d0cf88cb2e","Type":"ContainerStarted","Data":"9fa6376f3ba083f60f25e75401533db4602619bc61d4d79e66bb2b6005d0ff7f"} Nov 03 20:10:01.244558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:01.244084 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" event={"ID":"a3172c55-00d5-4500-b74e-e0d0cf88cb2e","Type":"ContainerStarted","Data":"12917f41f64b6e44fa4671cd1f087fdb23955d3c1324ad8ae3a580a7735bbb0a"} Nov 03 20:10:01.294556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:01.294130 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" podStartSLOduration=2.294110306 podStartE2EDuration="2.294110306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:01.287580495 +0000 UTC m=+1831.249958966" watchObservedRunningTime="2026-11-03 20:10:01.294110306 +0000 UTC m=+1831.256488746" Nov 03 20:10:01.754046 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:01.750889 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:10:01.756731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:01.756696 5111 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:10:01.757047 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:01.756996 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" podUID="a3172c55-00d5-4500-b74e-e0d0cf88cb2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 20:10:02.756063 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:02.756020 5111 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:10:02.756783 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:02.756749 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" podUID="a3172c55-00d5-4500-b74e-e0d0cf88cb2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 20:10:03.754052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:03.754011 5111 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:10:03.754378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:03.754346 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" podUID="a3172c55-00d5-4500-b74e-e0d0cf88cb2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 20:10:04.751553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:04.751208 5111 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:10:04.751553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:04.751273 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" podUID="a3172c55-00d5-4500-b74e-e0d0cf88cb2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 20:10:05.185558 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:10:05.180734 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:10:05.767211 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:05.765631 5111 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:10:05.767211 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:05.765705 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" podUID="a3172c55-00d5-4500-b74e-e0d0cf88cb2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 20:10:06.769001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:06.768957 5111 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:10:06.775666 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:06.775621 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" podUID="a3172c55-00d5-4500-b74e-e0d0cf88cb2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 20:10:07.024695 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:07.024611 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs"] Nov 03 20:10:07.030818 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:07.030787 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb465508-b9rqs"] Nov 03 20:10:07.750597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:07.750552 5111 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:10:07.750782 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:07.750613 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" podUID="a3172c55-00d5-4500-b74e-e0d0cf88cb2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 20:10:08.071664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.071581 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg"] Nov 03 20:10:08.097732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.097700 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd93879d-n4vhg"] Nov 03 20:10:08.108454 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.108412 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx"] Nov 03 20:10:08.115940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.115915 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.120246 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.119625 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-a476f8-openshift-default-dockercfg-lsx9m\"" Nov 03 20:10:08.121338 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.121304 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx"] Nov 03 20:10:08.317652 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.291856 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.352075 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.343686 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.355438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.355405 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.371575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.356235 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.371851 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.371780 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027fe01b-c13b-4f05-babf-c440793dfa7d-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.371955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.371918 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.372075 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.372054 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.372158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.372139 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.372220 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.372169 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6w7r\" (UniqueName: \"kubernetes.io/projected/027fe01b-c13b-4f05-babf-c440793dfa7d-kube-api-access-z6w7r\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.420555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.420187 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c86eb12-6efd-4389-9e92-7a5604ceea06" path="/var/lib/kubelet/pods/9c86eb12-6efd-4389-9e92-7a5604ceea06/volumes" Nov 03 20:10:08.424564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.420990 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b664f5-92aa-4853-8335-25d0d6692a5c" path="/var/lib/kubelet/pods/f8b664f5-92aa-4853-8335-25d0d6692a5c/volumes" Nov 03 20:10:08.500561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.493641 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027fe01b-c13b-4f05-babf-c440793dfa7d-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.500561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.493741 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.500561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.493842 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.500561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.493904 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.500561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.493933 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6w7r\" (UniqueName: \"kubernetes.io/projected/027fe01b-c13b-4f05-babf-c440793dfa7d-kube-api-access-z6w7r\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.500561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.493994 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.500561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.494167 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.500561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.494310 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.500561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.494513 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.509413 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.509361 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.510545 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.510506 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.512350 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.511784 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.512594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.512578 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.521959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.521936 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027fe01b-c13b-4f05-babf-c440793dfa7d-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.534554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.529267 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.534554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.529526 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.550551 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.536676 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6w7r\" (UniqueName: \"kubernetes.io/projected/027fe01b-c13b-4f05-babf-c440793dfa7d-kube-api-access-z6w7r\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.563554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.558625 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.754702 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.754666 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:08.756407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.756383 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:10:08.756790 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.756762 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:10:08.757661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:08.757641 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:10:09.095691 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.095570 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx"] Nov 03 20:10:09.122987 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.122482 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:09.123212 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.123186 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:09.123352 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.123335 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:09.197985 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.197957 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s"] Nov 03 20:10:09.206125 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.206101 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" Nov 03 20:10:09.210878 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.210830 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:10:09.210878 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.210830 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:10:09.211096 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.210933 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s"] Nov 03 20:10:09.221461 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.221224 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:10:09.221619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.221463 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:10:09.328281 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.328186 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b2wt\" (UniqueName: \"kubernetes.io/projected/c9c246c3-6ff6-458f-9d3b-e4d833e58595-kube-api-access-9b2wt\") pod \"maas-api-revoke-keys-e2e-api-a-d441dd-kc29s\" (UID: \"c9c246c3-6ff6-458f-9d3b-e4d833e58595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" Nov 03 20:10:09.328872 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.328831 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c246c3-6ff6-458f-9d3b-e4d833e58595-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d441dd-kc29s\" (UID: \"c9c246c3-6ff6-458f-9d3b-e4d833e58595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" Nov 03 20:10:09.329017 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.328916 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c246c3-6ff6-458f-9d3b-e4d833e58595-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d441dd-kc29s\" (UID: \"c9c246c3-6ff6-458f-9d3b-e4d833e58595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" Nov 03 20:10:09.430753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.430716 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b2wt\" (UniqueName: \"kubernetes.io/projected/c9c246c3-6ff6-458f-9d3b-e4d833e58595-kube-api-access-9b2wt\") pod \"maas-api-revoke-keys-e2e-api-a-d441dd-kc29s\" (UID: \"c9c246c3-6ff6-458f-9d3b-e4d833e58595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" Nov 03 20:10:09.430954 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.430940 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c246c3-6ff6-458f-9d3b-e4d833e58595-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d441dd-kc29s\" (UID: \"c9c246c3-6ff6-458f-9d3b-e4d833e58595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" Nov 03 20:10:09.431024 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.430993 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c246c3-6ff6-458f-9d3b-e4d833e58595-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d441dd-kc29s\" (UID: \"c9c246c3-6ff6-458f-9d3b-e4d833e58595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" Nov 03 20:10:09.431674 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.431646 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c246c3-6ff6-458f-9d3b-e4d833e58595-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d441dd-kc29s\" (UID: \"c9c246c3-6ff6-458f-9d3b-e4d833e58595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" Nov 03 20:10:09.433801 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.433769 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c246c3-6ff6-458f-9d3b-e4d833e58595-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d441dd-kc29s\" (UID: \"c9c246c3-6ff6-458f-9d3b-e4d833e58595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" Nov 03 20:10:09.438440 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.438416 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b2wt\" (UniqueName: \"kubernetes.io/projected/c9c246c3-6ff6-458f-9d3b-e4d833e58595-kube-api-access-9b2wt\") pod \"maas-api-revoke-keys-e2e-api-a-d441dd-kc29s\" (UID: \"c9c246c3-6ff6-458f-9d3b-e4d833e58595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" Nov 03 20:10:09.554469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.554435 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" Nov 03 20:10:09.663457 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.663345 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" event={"ID":"027fe01b-c13b-4f05-babf-c440793dfa7d","Type":"ContainerStarted","Data":"dfa0cc2e93fd859038ce9bd941bc694d22671bbd001e35bf883153a042ff2b19"} Nov 03 20:10:09.663457 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.663415 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" event={"ID":"027fe01b-c13b-4f05-babf-c440793dfa7d","Type":"ContainerStarted","Data":"5fe9c4b40527d7716e757518dd26c4a4e3600745649f91d17fc7979c7016afae"} Nov 03 20:10:09.691161 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.691060 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" podStartSLOduration=1.691040788 podStartE2EDuration="1.691040788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:09.68225958 +0000 UTC m=+1839.644638021" watchObservedRunningTime="2026-11-03 20:10:09.691040788 +0000 UTC m=+1839.653419229" Nov 03 20:10:09.755331 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.755287 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:09.897959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:09.897924 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s"] Nov 03 20:10:09.913994 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:10:09.913951 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c246c3_6ff6_458f_9d3b_e4d833e58595.slice/crio-5cec76cb86f158ba502375548958a2728732000e6ada2b08b2431697f3d217e4 WatchSource:0}: Error finding container 5cec76cb86f158ba502375548958a2728732000e6ada2b08b2431697f3d217e4: Status 404 returned error can't find the container with id 5cec76cb86f158ba502375548958a2728732000e6ada2b08b2431697f3d217e4 Nov 03 20:10:10.672830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:10.672798 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" event={"ID":"c9c246c3-6ff6-458f-9d3b-e4d833e58595","Type":"ContainerStarted","Data":"e7fd613f0c302b7fffabe6f0df8d24c704f83a581b4f110f5512a573aa568350"} Nov 03 20:10:10.673226 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:10.672852 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" event={"ID":"c9c246c3-6ff6-458f-9d3b-e4d833e58595","Type":"ContainerStarted","Data":"5cec76cb86f158ba502375548958a2728732000e6ada2b08b2431697f3d217e4"} Nov 03 20:10:10.693582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:10.693508 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" podStartSLOduration=1.693492005 podStartE2EDuration="1.693492005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:10.688176073 +0000 UTC m=+1840.650554512" watchObservedRunningTime="2026-11-03 20:10:10.693492005 +0000 UTC m=+1840.655870444" Nov 03 20:10:10.786610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:10.786383 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:11.705258 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:11.705218 5111 generic.go:356] "Generic (PLEG): container finished" podID="c9c246c3-6ff6-458f-9d3b-e4d833e58595" containerID="e7fd613f0c302b7fffabe6f0df8d24c704f83a581b4f110f5512a573aa568350" exitCode=0 Nov 03 20:10:11.705906 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:11.705499 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" event={"ID":"c9c246c3-6ff6-458f-9d3b-e4d833e58595","Type":"ContainerDied","Data":"e7fd613f0c302b7fffabe6f0df8d24c704f83a581b4f110f5512a573aa568350"} Nov 03 20:10:11.706172 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:11.706148 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:11.707182 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:11.707161 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:10:13.082910 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.082879 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f"] Nov 03 20:10:13.096978 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.096891 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00198b38-kcx4f"] Nov 03 20:10:13.175381 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.175354 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" Nov 03 20:10:13.237043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.236040 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c246c3-6ff6-458f-9d3b-e4d833e58595-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c246c3-6ff6-458f-9d3b-e4d833e58595-maas-api-xks-ca\") pod \"c9c246c3-6ff6-458f-9d3b-e4d833e58595\" (UID: \"c9c246c3-6ff6-458f-9d3b-e4d833e58595\") " Nov 03 20:10:13.237043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.236137 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c246c3-6ff6-458f-9d3b-e4d833e58595-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c246c3-6ff6-458f-9d3b-e4d833e58595-maas-api-openshift-ca\") pod \"c9c246c3-6ff6-458f-9d3b-e4d833e58595\" (UID: \"c9c246c3-6ff6-458f-9d3b-e4d833e58595\") " Nov 03 20:10:13.237043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.236207 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c246c3-6ff6-458f-9d3b-e4d833e58595-kube-api-access-9b2wt\" (UniqueName: \"kubernetes.io/projected/c9c246c3-6ff6-458f-9d3b-e4d833e58595-kube-api-access-9b2wt\") pod \"c9c246c3-6ff6-458f-9d3b-e4d833e58595\" (UID: \"c9c246c3-6ff6-458f-9d3b-e4d833e58595\") " Nov 03 20:10:13.238154 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.238082 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c246c3-6ff6-458f-9d3b-e4d833e58595-maas-api-openshift-ca" pod "c9c246c3-6ff6-458f-9d3b-e4d833e58595" (UID: "c9c246c3-6ff6-458f-9d3b-e4d833e58595"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:10:13.246254 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.242688 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c246c3-6ff6-458f-9d3b-e4d833e58595-maas-api-xks-ca" pod "c9c246c3-6ff6-458f-9d3b-e4d833e58595" (UID: "c9c246c3-6ff6-458f-9d3b-e4d833e58595"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:10:13.246254 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.244103 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c246c3-6ff6-458f-9d3b-e4d833e58595-kube-api-access-9b2wt" pod "c9c246c3-6ff6-458f-9d3b-e4d833e58595" (UID: "c9c246c3-6ff6-458f-9d3b-e4d833e58595"). InnerVolumeSpecName "kube-api-access-9b2wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:10:13.345555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.340442 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c246c3-6ff6-458f-9d3b-e4d833e58595-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:13.345555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.340478 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c246c3-6ff6-458f-9d3b-e4d833e58595-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:13.345555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.340493 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b2wt\" (UniqueName: \"kubernetes.io/projected/c9c246c3-6ff6-458f-9d3b-e4d833e58595-kube-api-access-9b2wt\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:13.533558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.532828 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp"] Nov 03 20:10:13.539658 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.534225 5111 cpu_manager.go:414] "containerMap: removing container" podUID="c9c246c3-6ff6-458f-9d3b-e4d833e58595" containerName="revoke-keys" Nov 03 20:10:13.539658 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.534256 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c246c3-6ff6-458f-9d3b-e4d833e58595" containerName="revoke-keys" Nov 03 20:10:13.539658 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.534570 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c246c3-6ff6-458f-9d3b-e4d833e58595" containerName="revoke-keys" Nov 03 20:10:13.540813 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.540148 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" Nov 03 20:10:13.588557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.587026 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-a476f8\"" Nov 03 20:10:13.588557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.587188 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-a476f8-dockercfg-sx7cs\"" Nov 03 20:10:13.673240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.662590 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp"] Nov 03 20:10:13.673240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.664435 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24d736c-014e-45e0-9459-bd36e2556f15-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp\" (UID: \"c24d736c-014e-45e0-9459-bd36e2556f15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" Nov 03 20:10:13.673240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.664476 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdcf\" (UniqueName: \"kubernetes.io/projected/c24d736c-014e-45e0-9459-bd36e2556f15-kube-api-access-gjdcf\") pod \"payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp\" (UID: \"c24d736c-014e-45e0-9459-bd36e2556f15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" Nov 03 20:10:13.886555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.883467 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24d736c-014e-45e0-9459-bd36e2556f15-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp\" (UID: \"c24d736c-014e-45e0-9459-bd36e2556f15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" Nov 03 20:10:13.886555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.883517 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdcf\" (UniqueName: \"kubernetes.io/projected/c24d736c-014e-45e0-9459-bd36e2556f15-kube-api-access-gjdcf\") pod \"payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp\" (UID: \"c24d736c-014e-45e0-9459-bd36e2556f15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" Nov 03 20:10:13.886555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:13.885238 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24d736c-014e-45e0-9459-bd36e2556f15-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp\" (UID: \"c24d736c-014e-45e0-9459-bd36e2556f15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" Nov 03 20:10:14.029630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:14.018399 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdcf\" (UniqueName: \"kubernetes.io/projected/c24d736c-014e-45e0-9459-bd36e2556f15-kube-api-access-gjdcf\") pod \"payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp\" (UID: \"c24d736c-014e-45e0-9459-bd36e2556f15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" Nov 03 20:10:14.188610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:14.187604 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" Nov 03 20:10:14.236603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:14.225377 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" Nov 03 20:10:14.253603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:14.250852 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a844b87-157c-4b4d-80f7-7b8bca7161f1" path="/var/lib/kubelet/pods/1a844b87-157c-4b4d-80f7-7b8bca7161f1/volumes" Nov 03 20:10:14.263646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:14.260013 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s" event={"ID":"c9c246c3-6ff6-458f-9d3b-e4d833e58595","Type":"ContainerDied","Data":"5cec76cb86f158ba502375548958a2728732000e6ada2b08b2431697f3d217e4"} Nov 03 20:10:14.263646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:14.260054 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cec76cb86f158ba502375548958a2728732000e6ada2b08b2431697f3d217e4" Nov 03 20:10:15.414172 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:15.414123 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp"] Nov 03 20:10:16.055616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:16.054596 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b"] Nov 03 20:10:16.110859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:16.110499 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2944544-x4s4b"] Nov 03 20:10:16.216555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:16.215189 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be1befb4-1127-459d-8a13-f4d71450c0a8" path="/var/lib/kubelet/pods/be1befb4-1127-459d-8a13-f4d71450c0a8/volumes" Nov 03 20:10:16.614557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:16.614190 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" event={"ID":"c24d736c-014e-45e0-9459-bd36e2556f15","Type":"ContainerStarted","Data":"199848a9c28a5e7c178afcb1634023e85dd9335367d3adafe46cc58b27ed605c"} Nov 03 20:10:16.614557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:16.614266 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" event={"ID":"c24d736c-014e-45e0-9459-bd36e2556f15","Type":"ContainerStarted","Data":"0df9b1c6c4eb0c7a5bd5e8462acf82747d9e49d2d5df75d055dcb7120edd1c0b"} Nov 03 20:10:16.618561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:16.615698 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" Nov 03 20:10:19.420516 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:19.420457 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" podStartSLOduration=6.420439875 podStartE2EDuration="6.420439875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:16.635697253 +0000 UTC m=+1846.598075691" watchObservedRunningTime="2026-11-03 20:10:19.420439875 +0000 UTC m=+1849.382818343" Nov 03 20:10:19.422033 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:19.422004 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp"] Nov 03 20:10:19.624577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:19.624527 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz"] Nov 03 20:10:20.950512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:20.950080 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:20.968549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:20.951202 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:10:20.968549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:20.967629 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:20.975104 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:20.975057 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:21.058700 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:21.058651 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:24.423290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:24.423238 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" podUID="c4cc53c8-f835-4d6c-aa61-7369d62ae1db" containerName="payload-processing" containerID="cri-o://09fa221b587eb8aa31b28a672fab43a9e44c979d970814dcd6b4180be677f5fb" gracePeriod=25 Nov 03 20:10:24.926200 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:24.801214 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" podUID="56cd2469-4b22-4aee-bf4e-a1daf2873729" containerName="payload-pre-processing" containerID="cri-o://c84d9bb6f0a029df16cb5679ed1d6f7e77b9d011e492b5cb52d55a239e8b85ec" gracePeriod=25 Nov 03 20:10:25.034554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:25.033729 5111 generic.go:356] "Generic (PLEG): container finished" podID="c4cc53c8-f835-4d6c-aa61-7369d62ae1db" containerID="09fa221b587eb8aa31b28a672fab43a9e44c979d970814dcd6b4180be677f5fb" exitCode=0 Nov 03 20:10:25.034554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:25.033968 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" event={"ID":"c4cc53c8-f835-4d6c-aa61-7369d62ae1db","Type":"ContainerDied","Data":"09fa221b587eb8aa31b28a672fab43a9e44c979d970814dcd6b4180be677f5fb"} Nov 03 20:10:26.055141 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.055103 5111 generic.go:356] "Generic (PLEG): container finished" podID="56cd2469-4b22-4aee-bf4e-a1daf2873729" containerID="c84d9bb6f0a029df16cb5679ed1d6f7e77b9d011e492b5cb52d55a239e8b85ec" exitCode=0 Nov 03 20:10:26.055653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.055266 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" event={"ID":"56cd2469-4b22-4aee-bf4e-a1daf2873729","Type":"ContainerDied","Data":"c84d9bb6f0a029df16cb5679ed1d6f7e77b9d011e492b5cb52d55a239e8b85ec"} Nov 03 20:10:26.055653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.055391 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" event={"ID":"56cd2469-4b22-4aee-bf4e-a1daf2873729","Type":"ContainerDied","Data":"74812fca06eb3272ee0c610793e2e38684efaed24da4f8e75caa19bcfe167daf"} Nov 03 20:10:26.055653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.055432 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74812fca06eb3272ee0c610793e2e38684efaed24da4f8e75caa19bcfe167daf" Nov 03 20:10:26.059340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.059313 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" Nov 03 20:10:26.114074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.114037 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" Nov 03 20:10:26.126348 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.125616 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4cc53c8-f835-4d6c-aa61-7369d62ae1db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4cc53c8-f835-4d6c-aa61-7369d62ae1db-plugins-config-volume\") pod \"c4cc53c8-f835-4d6c-aa61-7369d62ae1db\" (UID: \"c4cc53c8-f835-4d6c-aa61-7369d62ae1db\") " Nov 03 20:10:26.126348 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.125786 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4cc53c8-f835-4d6c-aa61-7369d62ae1db-kube-api-access-td54x\" (UniqueName: \"kubernetes.io/projected/c4cc53c8-f835-4d6c-aa61-7369d62ae1db-kube-api-access-td54x\") pod \"c4cc53c8-f835-4d6c-aa61-7369d62ae1db\" (UID: \"c4cc53c8-f835-4d6c-aa61-7369d62ae1db\") " Nov 03 20:10:26.126348 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.125982 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56cd2469-4b22-4aee-bf4e-a1daf2873729-kube-api-access-gdjr5\" (UniqueName: \"kubernetes.io/projected/56cd2469-4b22-4aee-bf4e-a1daf2873729-kube-api-access-gdjr5\") pod \"56cd2469-4b22-4aee-bf4e-a1daf2873729\" (UID: \"56cd2469-4b22-4aee-bf4e-a1daf2873729\") " Nov 03 20:10:26.126348 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.126034 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56cd2469-4b22-4aee-bf4e-a1daf2873729-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56cd2469-4b22-4aee-bf4e-a1daf2873729-plugins-config-volume\") pod \"56cd2469-4b22-4aee-bf4e-a1daf2873729\" (UID: \"56cd2469-4b22-4aee-bf4e-a1daf2873729\") " Nov 03 20:10:26.139468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.138840 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4cc53c8-f835-4d6c-aa61-7369d62ae1db-plugins-config-volume" pod "c4cc53c8-f835-4d6c-aa61-7369d62ae1db" (UID: "c4cc53c8-f835-4d6c-aa61-7369d62ae1db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:10:26.139468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.139106 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56cd2469-4b22-4aee-bf4e-a1daf2873729-plugins-config-volume" pod "56cd2469-4b22-4aee-bf4e-a1daf2873729" (UID: "56cd2469-4b22-4aee-bf4e-a1daf2873729"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:10:26.181152 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.181118 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4cc53c8-f835-4d6c-aa61-7369d62ae1db-kube-api-access-td54x" pod "c4cc53c8-f835-4d6c-aa61-7369d62ae1db" (UID: "c4cc53c8-f835-4d6c-aa61-7369d62ae1db"). InnerVolumeSpecName "kube-api-access-td54x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:10:26.181437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.181417 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56cd2469-4b22-4aee-bf4e-a1daf2873729-kube-api-access-gdjr5" pod "56cd2469-4b22-4aee-bf4e-a1daf2873729" (UID: "56cd2469-4b22-4aee-bf4e-a1daf2873729"). InnerVolumeSpecName "kube-api-access-gdjr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:10:26.229305 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.229275 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td54x\" (UniqueName: \"kubernetes.io/projected/c4cc53c8-f835-4d6c-aa61-7369d62ae1db-kube-api-access-td54x\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:26.229505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.229480 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdjr5\" (UniqueName: \"kubernetes.io/projected/56cd2469-4b22-4aee-bf4e-a1daf2873729-kube-api-access-gdjr5\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:26.229505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.229504 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56cd2469-4b22-4aee-bf4e-a1daf2873729-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:26.229715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:26.229516 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4cc53c8-f835-4d6c-aa61-7369d62ae1db-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:27.073903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:27.073868 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz" Nov 03 20:10:27.074703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:27.074674 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" Nov 03 20:10:27.075256 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:27.075231 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp" event={"ID":"c4cc53c8-f835-4d6c-aa61-7369d62ae1db","Type":"ContainerDied","Data":"827206bae471c0a78d19008b849fcd35e6d5467dc1cff87c0d67a80167a16014"} Nov 03 20:10:27.075362 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:27.075347 5111 scope.go:122] "RemoveContainer" containerID="09fa221b587eb8aa31b28a672fab43a9e44c979d970814dcd6b4180be677f5fb" Nov 03 20:10:27.110140 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:27.110109 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz"] Nov 03 20:10:27.116416 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:27.116390 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d441dd-64db594499-vhhfz"] Nov 03 20:10:27.158650 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:27.135401 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp"] Nov 03 20:10:27.169356 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:27.169314 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d441dd-6cdf67d9b-st7cp"] Nov 03 20:10:28.045124 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:28.045055 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4"] Nov 03 20:10:28.052289 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:28.052256 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edf9fa38-86cd4"] Nov 03 20:10:28.188779 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:28.188742 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56cd2469-4b22-4aee-bf4e-a1daf2873729" path="/var/lib/kubelet/pods/56cd2469-4b22-4aee-bf4e-a1daf2873729/volumes" Nov 03 20:10:28.189467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:28.189442 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65" path="/var/lib/kubelet/pods/5e82679a-ae4d-463d-9ca7-a7a4cd0ccd65/volumes" Nov 03 20:10:28.190123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:28.190100 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4cc53c8-f835-4d6c-aa61-7369d62ae1db" path="/var/lib/kubelet/pods/c4cc53c8-f835-4d6c-aa61-7369d62ae1db/volumes" Nov 03 20:10:28.840554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:28.840140 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" Nov 03 20:10:29.158746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:29.158711 5111 patch_prober.go:29] interesting pod/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:10:29.259976 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:29.259920 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:30.067617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.067578 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm"] Nov 03 20:10:30.069100 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.069071 5111 cpu_manager.go:414] "containerMap: removing container" podUID="56cd2469-4b22-4aee-bf4e-a1daf2873729" containerName="payload-pre-processing" Nov 03 20:10:30.069100 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.069100 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56cd2469-4b22-4aee-bf4e-a1daf2873729" containerName="payload-pre-processing" Nov 03 20:10:30.069263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.069192 5111 cpu_manager.go:414] "containerMap: removing container" podUID="c4cc53c8-f835-4d6c-aa61-7369d62ae1db" containerName="payload-processing" Nov 03 20:10:30.069263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.069201 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4cc53c8-f835-4d6c-aa61-7369d62ae1db" containerName="payload-processing" Nov 03 20:10:30.069484 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.069461 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4cc53c8-f835-4d6c-aa61-7369d62ae1db" containerName="payload-processing" Nov 03 20:10:30.069484 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.069500 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="56cd2469-4b22-4aee-bf4e-a1daf2873729" containerName="payload-pre-processing" Nov 03 20:10:30.131252 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.130773 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm"] Nov 03 20:10:30.132484 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.131582 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" Nov 03 20:10:30.137719 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.137619 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:10:30.138876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.138359 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:10:30.138876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.138500 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:10:30.138876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.138661 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:10:30.291011 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.290968 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-pggcm\" (UID: \"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" Nov 03 20:10:30.291425 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.291066 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-pggcm\" (UID: \"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" Nov 03 20:10:30.291425 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.291351 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pzqc\" (UniqueName: \"kubernetes.io/projected/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-kube-api-access-9pzqc\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-pggcm\" (UID: \"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" Nov 03 20:10:30.393249 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.393207 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pzqc\" (UniqueName: \"kubernetes.io/projected/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-kube-api-access-9pzqc\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-pggcm\" (UID: \"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" Nov 03 20:10:30.393496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.393451 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-pggcm\" (UID: \"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" Nov 03 20:10:30.393625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.393587 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-pggcm\" (UID: \"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" Nov 03 20:10:30.396219 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.396194 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:10:30.396732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.396716 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:10:30.402062 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.402039 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:10:30.404945 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.404910 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-pggcm\" (UID: \"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" Nov 03 20:10:30.436055 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.436018 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-pggcm\" (UID: \"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" Nov 03 20:10:30.436275 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.436252 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pzqc\" (UniqueName: \"kubernetes.io/projected/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-kube-api-access-9pzqc\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-pggcm\" (UID: \"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" Nov 03 20:10:30.555893 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.555861 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:10:30.563248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:30.563222 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" Nov 03 20:10:31.367591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:31.296606 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn"] Nov 03 20:10:31.408588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:31.405761 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff3be028-bz5kn"] Nov 03 20:10:31.425811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:31.425662 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x"] Nov 03 20:10:31.444357 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:31.443128 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x"] Nov 03 20:10:31.444357 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:31.443256 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" Nov 03 20:10:31.548941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:31.543377 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3484cb6-490f-409b-9a86-0ab06f1a4438-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02e1cb6c-6s62x\" (UID: \"f3484cb6-490f-409b-9a86-0ab06f1a4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" Nov 03 20:10:31.548941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:31.543594 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3484cb6-490f-409b-9a86-0ab06f1a4438-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02e1cb6c-6s62x\" (UID: \"f3484cb6-490f-409b-9a86-0ab06f1a4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" Nov 03 20:10:31.548941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:31.543800 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m84k\" (UniqueName: \"kubernetes.io/projected/f3484cb6-490f-409b-9a86-0ab06f1a4438-kube-api-access-8m84k\") pod \"maas-api-revoke-sub-02e1cb6c-6s62x\" (UID: \"f3484cb6-490f-409b-9a86-0ab06f1a4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" Nov 03 20:10:31.647481 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:31.647018 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3484cb6-490f-409b-9a86-0ab06f1a4438-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02e1cb6c-6s62x\" (UID: \"f3484cb6-490f-409b-9a86-0ab06f1a4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" Nov 03 20:10:31.647481 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:31.647241 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3484cb6-490f-409b-9a86-0ab06f1a4438-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02e1cb6c-6s62x\" (UID: \"f3484cb6-490f-409b-9a86-0ab06f1a4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" Nov 03 20:10:31.648733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:31.648661 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3484cb6-490f-409b-9a86-0ab06f1a4438-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02e1cb6c-6s62x\" (UID: \"f3484cb6-490f-409b-9a86-0ab06f1a4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" Nov 03 20:10:31.649908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:31.649020 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m84k\" (UniqueName: \"kubernetes.io/projected/f3484cb6-490f-409b-9a86-0ab06f1a4438-kube-api-access-8m84k\") pod \"maas-api-revoke-sub-02e1cb6c-6s62x\" (UID: \"f3484cb6-490f-409b-9a86-0ab06f1a4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" Nov 03 20:10:31.659173 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:31.656480 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3484cb6-490f-409b-9a86-0ab06f1a4438-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02e1cb6c-6s62x\" (UID: \"f3484cb6-490f-409b-9a86-0ab06f1a4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" Nov 03 20:10:31.689274 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:31.689244 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m84k\" (UniqueName: \"kubernetes.io/projected/f3484cb6-490f-409b-9a86-0ab06f1a4438-kube-api-access-8m84k\") pod \"maas-api-revoke-sub-02e1cb6c-6s62x\" (UID: \"f3484cb6-490f-409b-9a86-0ab06f1a4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" Nov 03 20:10:31.791459 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:31.790981 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm"] Nov 03 20:10:31.923476 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:31.923401 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" Nov 03 20:10:32.027984 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:32.027940 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947"] Nov 03 20:10:32.041831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:32.041797 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74f492cf-s7947"] Nov 03 20:10:32.193267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:32.193190 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0" path="/var/lib/kubelet/pods/17da1d9a-61f2-4f7e-a4d0-0904b72e6fd0/volumes" Nov 03 20:10:32.193841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:32.193818 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b" path="/var/lib/kubelet/pods/2961e7a6-1f7c-4ad3-b7e5-35c661b4af6b/volumes" Nov 03 20:10:32.337106 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:32.337064 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" event={"ID":"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8","Type":"ContainerStarted","Data":"d963de429593c09016b36d03ff7585f8a3113abf06dbe58ddcaec2d859f88be4"} Nov 03 20:10:32.367669 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:32.367631 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x"] Nov 03 20:10:32.380142 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:10:32.380112 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3484cb6_490f_409b_9a86_0ab06f1a4438.slice/crio-6a60bc9bb113acdd109314f27e5bb193d1f620ef82562afe75ec9839109d0672 WatchSource:0}: Error finding container 6a60bc9bb113acdd109314f27e5bb193d1f620ef82562afe75ec9839109d0672: Status 404 returned error can't find the container with id 6a60bc9bb113acdd109314f27e5bb193d1f620ef82562afe75ec9839109d0672 Nov 03 20:10:33.030503 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.030470 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh"] Nov 03 20:10:33.037244 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.037217 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6eb5149-k6xmh"] Nov 03 20:10:33.335654 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.331839 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s"] Nov 03 20:10:33.379366 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.379311 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" podStartSLOduration=4.379291913 podStartE2EDuration="4.379291913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:33.374046386 +0000 UTC m=+1863.336424825" watchObservedRunningTime="2026-11-03 20:10:33.379291913 +0000 UTC m=+1863.341670351" Nov 03 20:10:33.653770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.652521 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s"] Nov 03 20:10:33.653770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.652601 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" event={"ID":"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8","Type":"ContainerStarted","Data":"e556da01e315113b25cdbdac5e2a273fc45ddac0b3d36fea0aa03929cf33043f"} Nov 03 20:10:33.653770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.652655 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" event={"ID":"f3484cb6-490f-409b-9a86-0ab06f1a4438","Type":"ContainerStarted","Data":"965c7fa831155105c66e78ac75f683aee3773a5381c46131432b83e0632edb49"} Nov 03 20:10:33.653770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.652682 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" event={"ID":"f3484cb6-490f-409b-9a86-0ab06f1a4438","Type":"ContainerStarted","Data":"6a60bc9bb113acdd109314f27e5bb193d1f620ef82562afe75ec9839109d0672"} Nov 03 20:10:33.653770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.652786 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.751941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.751892 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3563a606-4dee-47cd-815f-ace2af65416b-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.752595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.751977 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.752595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.752092 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.752595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.752172 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.752595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.752207 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3563a606-4dee-47cd-815f-ace2af65416b-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.752595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.752240 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdrpf\" (UniqueName: \"kubernetes.io/projected/3563a606-4dee-47cd-815f-ace2af65416b-kube-api-access-fdrpf\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.752595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.752327 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.752595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.752506 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.752595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.752567 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3563a606-4dee-47cd-815f-ace2af65416b-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.854426 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.854390 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.854757 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.854732 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3563a606-4dee-47cd-815f-ace2af65416b-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.854993 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.854975 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3563a606-4dee-47cd-815f-ace2af65416b-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.855142 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.855128 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.855340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.855323 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.855511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.855495 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.855697 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.855678 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3563a606-4dee-47cd-815f-ace2af65416b-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.855809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.855794 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdrpf\" (UniqueName: \"kubernetes.io/projected/3563a606-4dee-47cd-815f-ace2af65416b-kube-api-access-fdrpf\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.855941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.855919 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3563a606-4dee-47cd-815f-ace2af65416b-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.856112 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.856093 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.856478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.856448 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.856587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.856469 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.856751 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.856727 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.856847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.856828 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.863193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.863167 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.863446 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.863421 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3563a606-4dee-47cd-815f-ace2af65416b-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.874079 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.874050 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdrpf\" (UniqueName: \"kubernetes.io/projected/3563a606-4dee-47cd-815f-ace2af65416b-kube-api-access-fdrpf\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.883187 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.883153 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3563a606-4dee-47cd-815f-ace2af65416b-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:33.994863 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:33.994821 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:34.209095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:34.208938 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596f1b95-fc67-4fea-bbb5-48c915d1fd2e" path="/var/lib/kubelet/pods/596f1b95-fc67-4fea-bbb5-48c915d1fd2e/volumes" Nov 03 20:10:34.478749 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:34.450575 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s"] Nov 03 20:10:34.510438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:34.510402 5111 generic.go:356] "Generic (PLEG): container finished" podID="9a21abc9-e2e3-4d49-917f-0277dfe0bcd8" containerID="e556da01e315113b25cdbdac5e2a273fc45ddac0b3d36fea0aa03929cf33043f" exitCode=0 Nov 03 20:10:34.511674 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:34.511646 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" event={"ID":"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8","Type":"ContainerDied","Data":"e556da01e315113b25cdbdac5e2a273fc45ddac0b3d36fea0aa03929cf33043f"} Nov 03 20:10:34.516921 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:34.516891 5111 generic.go:356] "Generic (PLEG): container finished" podID="f3484cb6-490f-409b-9a86-0ab06f1a4438" containerID="965c7fa831155105c66e78ac75f683aee3773a5381c46131432b83e0632edb49" exitCode=0 Nov 03 20:10:34.517262 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:34.517235 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" event={"ID":"f3484cb6-490f-409b-9a86-0ab06f1a4438","Type":"ContainerDied","Data":"965c7fa831155105c66e78ac75f683aee3773a5381c46131432b83e0632edb49"} Nov 03 20:10:34.523423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:34.523381 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:34.523989 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:34.523758 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:34.523989 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:34.523835 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:35.404333 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:10:35.404287 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:10:35.539574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:35.539496 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" event={"ID":"3563a606-4dee-47cd-815f-ace2af65416b","Type":"ContainerStarted","Data":"5d9eef1047bf66d368a9a2e1b58a1a8a6b349ccbabb3e5a928fcae319695d96e"} Nov 03 20:10:35.540142 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:35.539612 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" event={"ID":"3563a606-4dee-47cd-815f-ace2af65416b","Type":"ContainerStarted","Data":"5de821e627c2ff7c60a4d87df9cd3d0f83ba575d33c4125b6f623656f2c9936b"} Nov 03 20:10:35.575556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:35.571366 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" podStartSLOduration=2.571345601 podStartE2EDuration="2.571345601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:35.570073776 +0000 UTC m=+1865.532452213" watchObservedRunningTime="2026-11-03 20:10:35.571345601 +0000 UTC m=+1865.533724041" Nov 03 20:10:35.913460 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:35.913025 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" Nov 03 20:10:36.003970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:35.995515 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:36.003970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:35.996915 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 03 20:10:36.003970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:35.996962 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" podUID="3563a606-4dee-47cd-815f-ace2af65416b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 03 20:10:36.059646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.029586 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd"] Nov 03 20:10:36.059646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.040577 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-maas-api-openshift-ca\") pod \"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8\" (UID: \"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8\") " Nov 03 20:10:36.059646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.040695 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-maas-api-xks-ca\") pod \"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8\" (UID: \"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8\") " Nov 03 20:10:36.059646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.040790 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-kube-api-access-9pzqc\" (UniqueName: \"kubernetes.io/projected/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-kube-api-access-9pzqc\") pod \"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8\" (UID: \"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8\") " Nov 03 20:10:36.059646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.044870 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-maas-api-openshift-ca" pod "9a21abc9-e2e3-4d49-917f-0277dfe0bcd8" (UID: "9a21abc9-e2e3-4d49-917f-0277dfe0bcd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:10:36.083563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.075784 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03cfbe52-98qzd"] Nov 03 20:10:36.087888 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.085940 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" Nov 03 20:10:36.092010 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.091960 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-kube-api-access-9pzqc" pod "9a21abc9-e2e3-4d49-917f-0277dfe0bcd8" (UID: "9a21abc9-e2e3-4d49-917f-0277dfe0bcd8"). InnerVolumeSpecName "kube-api-access-9pzqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:10:36.099550 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.098133 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-maas-api-xks-ca" pod "9a21abc9-e2e3-4d49-917f-0277dfe0bcd8" (UID: "9a21abc9-e2e3-4d49-917f-0277dfe0bcd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:10:36.153895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.143140 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3484cb6-490f-409b-9a86-0ab06f1a4438-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3484cb6-490f-409b-9a86-0ab06f1a4438-maas-api-xks-ca\") pod \"f3484cb6-490f-409b-9a86-0ab06f1a4438\" (UID: \"f3484cb6-490f-409b-9a86-0ab06f1a4438\") " Nov 03 20:10:36.153895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.143220 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3484cb6-490f-409b-9a86-0ab06f1a4438-kube-api-access-8m84k\" (UniqueName: \"kubernetes.io/projected/f3484cb6-490f-409b-9a86-0ab06f1a4438-kube-api-access-8m84k\") pod \"f3484cb6-490f-409b-9a86-0ab06f1a4438\" (UID: \"f3484cb6-490f-409b-9a86-0ab06f1a4438\") " Nov 03 20:10:36.153895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.143447 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3484cb6-490f-409b-9a86-0ab06f1a4438-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3484cb6-490f-409b-9a86-0ab06f1a4438-maas-api-openshift-ca\") pod \"f3484cb6-490f-409b-9a86-0ab06f1a4438\" (UID: \"f3484cb6-490f-409b-9a86-0ab06f1a4438\") " Nov 03 20:10:36.153895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.144387 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:36.153895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.144413 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:36.153895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.144426 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pzqc\" (UniqueName: \"kubernetes.io/projected/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8-kube-api-access-9pzqc\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:36.175558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.156912 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3484cb6-490f-409b-9a86-0ab06f1a4438-maas-api-openshift-ca" pod "f3484cb6-490f-409b-9a86-0ab06f1a4438" (UID: "f3484cb6-490f-409b-9a86-0ab06f1a4438"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:10:36.181606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.181568 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3484cb6-490f-409b-9a86-0ab06f1a4438-maas-api-xks-ca" pod "f3484cb6-490f-409b-9a86-0ab06f1a4438" (UID: "f3484cb6-490f-409b-9a86-0ab06f1a4438"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:10:36.185096 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.185062 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3484cb6-490f-409b-9a86-0ab06f1a4438-kube-api-access-8m84k" pod "f3484cb6-490f-409b-9a86-0ab06f1a4438" (UID: "f3484cb6-490f-409b-9a86-0ab06f1a4438"). InnerVolumeSpecName "kube-api-access-8m84k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:10:36.195254 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.195221 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2548d1ba-2355-4d85-8e21-2938cefbbce4" path="/var/lib/kubelet/pods/2548d1ba-2355-4d85-8e21-2938cefbbce4/volumes" Nov 03 20:10:36.269631 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.268326 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3484cb6-490f-409b-9a86-0ab06f1a4438-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:36.269631 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.268365 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3484cb6-490f-409b-9a86-0ab06f1a4438-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:36.269631 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.268386 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m84k\" (UniqueName: \"kubernetes.io/projected/f3484cb6-490f-409b-9a86-0ab06f1a4438-kube-api-access-8m84k\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:36.601639 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.601555 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" Nov 03 20:10:36.602116 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.601668 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x" event={"ID":"f3484cb6-490f-409b-9a86-0ab06f1a4438","Type":"ContainerDied","Data":"6a60bc9bb113acdd109314f27e5bb193d1f620ef82562afe75ec9839109d0672"} Nov 03 20:10:36.602116 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.601730 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a60bc9bb113acdd109314f27e5bb193d1f620ef82562afe75ec9839109d0672" Nov 03 20:10:36.604811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.604339 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" Nov 03 20:10:36.604811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.604449 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm" event={"ID":"9a21abc9-e2e3-4d49-917f-0277dfe0bcd8","Type":"ContainerDied","Data":"d963de429593c09016b36d03ff7585f8a3113abf06dbe58ddcaec2d859f88be4"} Nov 03 20:10:36.604811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:36.604500 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d963de429593c09016b36d03ff7585f8a3113abf06dbe58ddcaec2d859f88be4" Nov 03 20:10:37.007576 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:37.007505 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 03 20:10:37.007845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:37.007581 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" podUID="3563a606-4dee-47cd-815f-ace2af65416b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 03 20:10:37.609554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:37.605673 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:10:37.609554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:37.605754 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" podUID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:38.002640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:38.002578 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 03 20:10:38.002846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:38.002662 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" podUID="3563a606-4dee-47cd-815f-ace2af65416b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 03 20:10:38.995722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:38.995684 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 03 20:10:38.996138 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:38.995753 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" podUID="3563a606-4dee-47cd-815f-ace2af65416b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 03 20:10:39.781182 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:39.781149 5111 generic.go:356] "Generic (PLEG): container finished" podID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerID="aa025bd892cda3c538628a7c854d8a16a862172523c2d43a2b1ab3209f664e23" exitCode=0 Nov 03 20:10:39.781406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:39.781219 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" event={"ID":"17938ed6-2f40-4c8a-84f4-97bda29142cc","Type":"ContainerDied","Data":"aa025bd892cda3c538628a7c854d8a16a862172523c2d43a2b1ab3209f664e23"} Nov 03 20:10:39.781773 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:39.781755 5111 scope.go:122] "RemoveContainer" containerID="aa025bd892cda3c538628a7c854d8a16a862172523c2d43a2b1ab3209f664e23" Nov 03 20:10:39.783870 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:39.783843 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:39.783968 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:39.783911 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:39.784027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:39.783961 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:39.995620 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:39.995590 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 03 20:10:39.995757 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:39.995651 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" podUID="3563a606-4dee-47cd-815f-ace2af65416b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 03 20:10:40.928616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:40.899814 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" event={"ID":"17938ed6-2f40-4c8a-84f4-97bda29142cc","Type":"ContainerStarted","Data":"f3749975f46f05c57663e40bcd88039b67509c020b4e6748c4eb7b558075e9dd"} Nov 03 20:10:40.997397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:40.996935 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 03 20:10:40.997397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:40.996999 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" podUID="3563a606-4dee-47cd-815f-ace2af65416b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 03 20:10:41.212816 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:41.212695 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86"] Nov 03 20:10:41.225359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:41.225290 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25afd63c-h7p86"] Nov 03 20:10:41.392264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:41.392173 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:10:41.392473 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:41.392408 5111 patch_prober.go:29] interesting pod/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 20:10:41.392473 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:41.392454 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 20:10:42.003589 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:42.003556 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:42.004618 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:42.004597 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:42.005454 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:42.005431 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:10:42.486371 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:42.486327 5111 patch_prober.go:29] interesting pod/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 20:10:42.486741 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:42.486695 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 20:10:42.589558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:42.587666 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54" path="/var/lib/kubelet/pods/15f7f1c5-5b98-4141-98c2-2b0a8f1b8d54/volumes" Nov 03 20:10:43.116637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:43.116601 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:10:43.379282 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:43.379240 5111 patch_prober.go:29] interesting pod/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 20:10:43.379477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:43.379311 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 20:10:44.379773 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:44.379733 5111 patch_prober.go:29] interesting pod/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 20:10:44.385735 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:44.385686 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 20:10:44.601289 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:10:44.601258 5111 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de3942d_307c_4f4c_b291_2b69ad18f472.slice/crio-conmon-1e8d76fd89609448f8f6a2d4f962288558792de0ccea2dc41e0e394942b9f617.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:10:44.601827 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:10:44.601806 5111 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de3942d_307c_4f4c_b291_2b69ad18f472.slice/crio-conmon-1e8d76fd89609448f8f6a2d4f962288558792de0ccea2dc41e0e394942b9f617.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:10:44.933027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:44.932994 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5"] Nov 03 20:10:44.933850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:44.933835 5111 cpu_manager.go:414] "containerMap: removing container" podUID="f3484cb6-490f-409b-9a86-0ab06f1a4438" containerName="revoke-keys" Nov 03 20:10:44.933902 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:44.933852 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3484cb6-490f-409b-9a86-0ab06f1a4438" containerName="revoke-keys" Nov 03 20:10:44.933902 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:44.933869 5111 cpu_manager.go:414] "containerMap: removing container" podUID="9a21abc9-e2e3-4d49-917f-0277dfe0bcd8" containerName="revoke-keys" Nov 03 20:10:44.933902 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:44.933875 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a21abc9-e2e3-4d49-917f-0277dfe0bcd8" containerName="revoke-keys" Nov 03 20:10:44.934074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:44.934063 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3484cb6-490f-409b-9a86-0ab06f1a4438" containerName="revoke-keys" Nov 03 20:10:44.934109 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:44.934076 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a21abc9-e2e3-4d49-917f-0277dfe0bcd8" containerName="revoke-keys" Nov 03 20:10:44.937225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:44.937205 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" Nov 03 20:10:44.942183 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:44.942158 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:10:44.969567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:44.962577 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5"] Nov 03 20:10:44.994454 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:44.994420 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:10:44.995817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:44.995408 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:10:44.995817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:44.995600 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:10:45.007026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.006966 5111 generic.go:356] "Generic (PLEG): container finished" podID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerID="1e8d76fd89609448f8f6a2d4f962288558792de0ccea2dc41e0e394942b9f617" exitCode=0 Nov 03 20:10:45.007571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.007187 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" event={"ID":"7de3942d-307c-4f4c-b291-2b69ad18f472","Type":"ContainerDied","Data":"1e8d76fd89609448f8f6a2d4f962288558792de0ccea2dc41e0e394942b9f617"} Nov 03 20:10:45.008267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.008248 5111 scope.go:122] "RemoveContainer" containerID="1e8d76fd89609448f8f6a2d4f962288558792de0ccea2dc41e0e394942b9f617" Nov 03 20:10:45.011377 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.011344 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:45.011599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.011573 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:45.011807 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.011785 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:45.026687 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.026409 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d"] Nov 03 20:10:45.080552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.066051 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a654e1-f4g8d"] Nov 03 20:10:45.092226 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.092193 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft52k\" (UniqueName: \"kubernetes.io/projected/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-kube-api-access-ft52k\") pod \"maas-api-revoke-sub-8d78eb9a-qmcj5\" (UID: \"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" Nov 03 20:10:45.092582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.092556 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d78eb9a-qmcj5\" (UID: \"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" Nov 03 20:10:45.093392 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.093367 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d78eb9a-qmcj5\" (UID: \"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" Nov 03 20:10:45.204952 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.203959 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft52k\" (UniqueName: \"kubernetes.io/projected/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-kube-api-access-ft52k\") pod \"maas-api-revoke-sub-8d78eb9a-qmcj5\" (UID: \"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" Nov 03 20:10:45.204952 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.204088 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d78eb9a-qmcj5\" (UID: \"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" Nov 03 20:10:45.204952 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.204351 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d78eb9a-qmcj5\" (UID: \"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" Nov 03 20:10:45.205249 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.205129 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d78eb9a-qmcj5\" (UID: \"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" Nov 03 20:10:45.209955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.209884 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d78eb9a-qmcj5\" (UID: \"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" Nov 03 20:10:45.215993 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.215970 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft52k\" (UniqueName: \"kubernetes.io/projected/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-kube-api-access-ft52k\") pod \"maas-api-revoke-sub-8d78eb9a-qmcj5\" (UID: \"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" Nov 03 20:10:45.370599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.365590 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" Nov 03 20:10:45.403105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.382910 5111 patch_prober.go:29] interesting pod/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 20:10:45.403105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:45.382964 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 20:10:46.008083 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:46.008054 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5"] Nov 03 20:10:46.041211 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:46.041176 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" event={"ID":"7de3942d-307c-4f4c-b291-2b69ad18f472","Type":"ContainerStarted","Data":"42ac199a7bb1fa7a351c74c38ae39ac877081c64bcebb1f417ab7b939396fd8a"} Nov 03 20:10:46.043714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:46.043686 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv"] Nov 03 20:10:46.044224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:46.044198 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" event={"ID":"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c","Type":"ContainerStarted","Data":"f37b4ef560b134670d32ab7b600cd9e322da01705b8eb1772390c3b23ced360a"} Nov 03 20:10:46.089977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:46.085503 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d7663c-r5wdv"] Nov 03 20:10:46.401566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:46.380658 5111 patch_prober.go:29] interesting pod/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 20:10:46.401566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:46.380716 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 20:10:46.581599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:46.578316 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f" path="/var/lib/kubelet/pods/7ee7b794-4f05-4e4f-8b1d-3a6994f32a7f/volumes" Nov 03 20:10:46.581599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:46.579342 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d76d5576-0a53-4653-bb66-c8e640e6f819" path="/var/lib/kubelet/pods/d76d5576-0a53-4653-bb66-c8e640e6f819/volumes" Nov 03 20:10:46.581599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:46.579903 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s"] Nov 03 20:10:46.618740 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:46.618702 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d441dd-kc29s"] Nov 03 20:10:46.897558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:46.889292 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:10:46.897558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:46.894975 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:10:46.897558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:46.895046 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" podUID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 20:10:47.140437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.140403 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7"] Nov 03 20:10:47.204823 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.204735 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7" Nov 03 20:10:47.212662 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.211523 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" event={"ID":"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c","Type":"ContainerStarted","Data":"78a6846a729f3b96c18b1bb9ae407b86170bba31fb01306ed66f64f26608745e"} Nov 03 20:10:47.217767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.217745 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7"] Nov 03 20:10:47.224578 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.224511 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll"] Nov 03 20:10:47.229961 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.229935 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6274d31-fv6ll"] Nov 03 20:10:47.236659 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.236636 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7\" (UID: \"a0d50e44-d18a-4573-a9ea-c86ef7f1a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7" Nov 03 20:10:47.236775 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.236681 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbsfq\" (UniqueName: \"kubernetes.io/projected/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-kube-api-access-wbsfq\") pod \"maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7\" (UID: \"a0d50e44-d18a-4573-a9ea-c86ef7f1a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7" Nov 03 20:10:47.236775 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.236765 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7\" (UID: \"a0d50e44-d18a-4573-a9ea-c86ef7f1a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7" Nov 03 20:10:47.315881 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.315760 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" podStartSLOduration=3.3157396439999998 podStartE2EDuration="3.315739644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:47.309435786 +0000 UTC m=+1877.271814227" watchObservedRunningTime="2026-11-03 20:10:47.315739644 +0000 UTC m=+1877.278118084" Nov 03 20:10:47.338151 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.338115 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7\" (UID: \"a0d50e44-d18a-4573-a9ea-c86ef7f1a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7" Nov 03 20:10:47.338348 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.338162 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbsfq\" (UniqueName: \"kubernetes.io/projected/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-kube-api-access-wbsfq\") pod \"maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7\" (UID: \"a0d50e44-d18a-4573-a9ea-c86ef7f1a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7" Nov 03 20:10:47.338348 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.338288 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7\" (UID: \"a0d50e44-d18a-4573-a9ea-c86ef7f1a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7" Nov 03 20:10:47.339466 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.339437 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7\" (UID: \"a0d50e44-d18a-4573-a9ea-c86ef7f1a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7" Nov 03 20:10:47.341203 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.341173 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7\" (UID: \"a0d50e44-d18a-4573-a9ea-c86ef7f1a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7" Nov 03 20:10:47.372578 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.372322 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbsfq\" (UniqueName: \"kubernetes.io/projected/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-kube-api-access-wbsfq\") pod \"maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7\" (UID: \"a0d50e44-d18a-4573-a9ea-c86ef7f1a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7" Nov 03 20:10:47.381573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.380003 5111 patch_prober.go:29] interesting pod/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 20:10:47.381573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.380067 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 20:10:47.525274 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.525244 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7" Nov 03 20:10:47.878334 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.878303 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:10:47.878829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.878357 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" podUID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 20:10:47.883218 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:47.883186 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7"] Nov 03 20:10:47.885110 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:10:47.885086 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d50e44_d18a_4573_a9ea_c86ef7f1a227.slice/crio-575e74b8a73b3ea747fa01ad1a48f9309af37ef75ac8730afc998f220f37fa17 WatchSource:0}: Error finding container 575e74b8a73b3ea747fa01ad1a48f9309af37ef75ac8730afc998f220f37fa17: Status 404 returned error can't find the container with id 575e74b8a73b3ea747fa01ad1a48f9309af37ef75ac8730afc998f220f37fa17 Nov 03 20:10:48.244481 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:48.244308 5111 generic.go:356] "Generic (PLEG): container finished" podID="6d60aeef-cb74-40e4-b1aa-42fd6ee7962c" containerID="78a6846a729f3b96c18b1bb9ae407b86170bba31fb01306ed66f64f26608745e" exitCode=0 Nov 03 20:10:48.250444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:48.244323 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c72ddb-a97a-43b1-b26a-7814772a7d4c" path="/var/lib/kubelet/pods/11c72ddb-a97a-43b1-b26a-7814772a7d4c/volumes" Nov 03 20:10:48.250444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:48.245492 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c246c3-6ff6-458f-9d3b-e4d833e58595" path="/var/lib/kubelet/pods/c9c246c3-6ff6-458f-9d3b-e4d833e58595/volumes" Nov 03 20:10:48.251287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:48.251259 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7" event={"ID":"a0d50e44-d18a-4573-a9ea-c86ef7f1a227","Type":"ContainerStarted","Data":"575e74b8a73b3ea747fa01ad1a48f9309af37ef75ac8730afc998f220f37fa17"} Nov 03 20:10:48.251598 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:48.251575 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" event={"ID":"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c","Type":"ContainerDied","Data":"78a6846a729f3b96c18b1bb9ae407b86170bba31fb01306ed66f64f26608745e"} Nov 03 20:10:48.383933 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:48.383908 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:10:48.385143 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:48.385126 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:10:48.871961 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:48.871917 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:10:48.872349 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:48.872317 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" podUID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 20:10:49.282420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:49.282334 5111 generic.go:356] "Generic (PLEG): container finished" podID="a0d50e44-d18a-4573-a9ea-c86ef7f1a227" containerID="79dbc5eda539c25bd3150a9a767896bf663a98b9f623f6172af4f0ddc4fd2f54" exitCode=0 Nov 03 20:10:49.282897 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:49.282427 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7" event={"ID":"a0d50e44-d18a-4573-a9ea-c86ef7f1a227","Type":"ContainerDied","Data":"79dbc5eda539c25bd3150a9a767896bf663a98b9f623f6172af4f0ddc4fd2f54"} Nov 03 20:10:49.527540 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:49.527497 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" Nov 03 20:10:49.676571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:49.667413 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-maas-api-xks-ca\") pod \"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c\" (UID: \"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c\") " Nov 03 20:10:49.676571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:49.667923 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-maas-api-openshift-ca\") pod \"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c\" (UID: \"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c\") " Nov 03 20:10:49.676571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:49.667991 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-kube-api-access-ft52k\" (UniqueName: \"kubernetes.io/projected/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-kube-api-access-ft52k\") pod \"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c\" (UID: \"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c\") " Nov 03 20:10:49.773593 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:49.743046 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-maas-api-xks-ca" pod "6d60aeef-cb74-40e4-b1aa-42fd6ee7962c" (UID: "6d60aeef-cb74-40e4-b1aa-42fd6ee7962c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:10:49.773944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:49.773740 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-kube-api-access-ft52k" pod "6d60aeef-cb74-40e4-b1aa-42fd6ee7962c" (UID: "6d60aeef-cb74-40e4-b1aa-42fd6ee7962c"). InnerVolumeSpecName "kube-api-access-ft52k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:10:49.788574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:49.774804 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft52k\" (UniqueName: \"kubernetes.io/projected/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-kube-api-access-ft52k\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:49.788574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:49.774835 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:49.788891 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:49.788860 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-maas-api-openshift-ca" pod "6d60aeef-cb74-40e4-b1aa-42fd6ee7962c" (UID: "6d60aeef-cb74-40e4-b1aa-42fd6ee7962c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:10:49.874460 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:49.872973 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:10:49.874460 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:49.873036 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" podUID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 20:10:49.877025 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:49.876997 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:50.297523 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:50.297493 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" Nov 03 20:10:50.298028 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:50.297492 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5" event={"ID":"6d60aeef-cb74-40e4-b1aa-42fd6ee7962c","Type":"ContainerDied","Data":"f37b4ef560b134670d32ab7b600cd9e322da01705b8eb1772390c3b23ced360a"} Nov 03 20:10:50.298028 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:50.297620 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f37b4ef560b134670d32ab7b600cd9e322da01705b8eb1772390c3b23ced360a" Nov 03 20:10:50.880111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:50.880078 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:10:50.880423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:50.880388 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" podUID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 20:10:50.982559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:50.969388 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7" Nov 03 20:10:51.073567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:51.072333 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-kube-api-access-wbsfq\" (UniqueName: \"kubernetes.io/projected/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-kube-api-access-wbsfq\") pod \"a0d50e44-d18a-4573-a9ea-c86ef7f1a227\" (UID: \"a0d50e44-d18a-4573-a9ea-c86ef7f1a227\") " Nov 03 20:10:51.073567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:51.072600 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-maas-api-openshift-ca\") pod \"a0d50e44-d18a-4573-a9ea-c86ef7f1a227\" (UID: \"a0d50e44-d18a-4573-a9ea-c86ef7f1a227\") " Nov 03 20:10:51.073567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:51.072743 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-maas-api-xks-ca\") pod \"a0d50e44-d18a-4573-a9ea-c86ef7f1a227\" (UID: \"a0d50e44-d18a-4573-a9ea-c86ef7f1a227\") " Nov 03 20:10:51.117521 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:51.077964 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-maas-api-openshift-ca" pod "a0d50e44-d18a-4573-a9ea-c86ef7f1a227" (UID: "a0d50e44-d18a-4573-a9ea-c86ef7f1a227"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:10:51.117521 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:51.109087 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-kube-api-access-wbsfq" pod "a0d50e44-d18a-4573-a9ea-c86ef7f1a227" (UID: "a0d50e44-d18a-4573-a9ea-c86ef7f1a227"). InnerVolumeSpecName "kube-api-access-wbsfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:10:51.135969 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:51.134701 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-maas-api-xks-ca" pod "a0d50e44-d18a-4573-a9ea-c86ef7f1a227" (UID: "a0d50e44-d18a-4573-a9ea-c86ef7f1a227"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:10:51.138592 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:51.138551 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbsfq\" (UniqueName: \"kubernetes.io/projected/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-kube-api-access-wbsfq\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:51.138592 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:51.138590 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:51.138592 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:51.138606 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d50e44-d18a-4573-a9ea-c86ef7f1a227-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:51.308998 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:51.308969 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7" Nov 03 20:10:51.309389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:51.309010 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7" event={"ID":"a0d50e44-d18a-4573-a9ea-c86ef7f1a227","Type":"ContainerDied","Data":"575e74b8a73b3ea747fa01ad1a48f9309af37ef75ac8730afc998f220f37fa17"} Nov 03 20:10:51.309389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:51.309059 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="575e74b8a73b3ea747fa01ad1a48f9309af37ef75ac8730afc998f220f37fa17" Nov 03 20:10:51.563609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:51.556012 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:10:51.875552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:51.873637 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:10:51.875552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:51.873712 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" podUID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 20:10:52.027971 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:52.027936 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw"] Nov 03 20:10:52.035944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:52.035909 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da6d9b3-554vw"] Nov 03 20:10:52.197132 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:52.197062 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c6efbc-ed2c-4664-b08b-1e8f97ac289a" path="/var/lib/kubelet/pods/11c6efbc-ed2c-4664-b08b-1e8f97ac289a/volumes" Nov 03 20:10:53.005245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.005213 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:10:53.012760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.012716 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:10:53.668554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.663116 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf"] Nov 03 20:10:53.668554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.664469 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6d60aeef-cb74-40e4-b1aa-42fd6ee7962c" containerName="revoke-keys" Nov 03 20:10:53.668554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.664492 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d60aeef-cb74-40e4-b1aa-42fd6ee7962c" containerName="revoke-keys" Nov 03 20:10:53.668554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.664558 5111 cpu_manager.go:414] "containerMap: removing container" podUID="a0d50e44-d18a-4573-a9ea-c86ef7f1a227" containerName="revoke-keys" Nov 03 20:10:53.668554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.664568 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d50e44-d18a-4573-a9ea-c86ef7f1a227" containerName="revoke-keys" Nov 03 20:10:53.668554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.664860 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d60aeef-cb74-40e4-b1aa-42fd6ee7962c" containerName="revoke-keys" Nov 03 20:10:53.668554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.664881 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d50e44-d18a-4573-a9ea-c86ef7f1a227" containerName="revoke-keys" Nov 03 20:10:53.669074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.669045 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" Nov 03 20:10:53.689477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.674897 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:10:53.689477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.674958 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:10:53.689477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.675070 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:10:53.689477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.675109 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:10:53.702018 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.696450 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb4dcd0-d282-43ba-919b-ccd559228fa9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf\" (UID: \"ebb4dcd0-d282-43ba-919b-ccd559228fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" Nov 03 20:10:53.702018 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.696566 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb4dcd0-d282-43ba-919b-ccd559228fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf\" (UID: \"ebb4dcd0-d282-43ba-919b-ccd559228fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" Nov 03 20:10:53.702018 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.696958 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcn69\" (UniqueName: \"kubernetes.io/projected/ebb4dcd0-d282-43ba-919b-ccd559228fa9-kube-api-access-lcn69\") pod \"maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf\" (UID: \"ebb4dcd0-d282-43ba-919b-ccd559228fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" Nov 03 20:10:53.702018 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.697570 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf"] Nov 03 20:10:53.798644 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.798583 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb4dcd0-d282-43ba-919b-ccd559228fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf\" (UID: \"ebb4dcd0-d282-43ba-919b-ccd559228fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" Nov 03 20:10:53.798955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.798932 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcn69\" (UniqueName: \"kubernetes.io/projected/ebb4dcd0-d282-43ba-919b-ccd559228fa9-kube-api-access-lcn69\") pod \"maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf\" (UID: \"ebb4dcd0-d282-43ba-919b-ccd559228fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" Nov 03 20:10:53.799071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.799057 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb4dcd0-d282-43ba-919b-ccd559228fa9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf\" (UID: \"ebb4dcd0-d282-43ba-919b-ccd559228fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" Nov 03 20:10:53.799272 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.799253 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb4dcd0-d282-43ba-919b-ccd559228fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf\" (UID: \"ebb4dcd0-d282-43ba-919b-ccd559228fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" Nov 03 20:10:53.801522 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.801504 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb4dcd0-d282-43ba-919b-ccd559228fa9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf\" (UID: \"ebb4dcd0-d282-43ba-919b-ccd559228fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" Nov 03 20:10:53.806995 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:53.806969 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcn69\" (UniqueName: \"kubernetes.io/projected/ebb4dcd0-d282-43ba-919b-ccd559228fa9-kube-api-access-lcn69\") pod \"maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf\" (UID: \"ebb4dcd0-d282-43ba-919b-ccd559228fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" Nov 03 20:10:54.026426 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:54.026345 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" Nov 03 20:10:54.332756 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:54.332721 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf"] Nov 03 20:10:54.333135 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:10:54.333106 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb4dcd0_d282_43ba_919b_ccd559228fa9.slice/crio-12f7be96c90451ce7f1f6a3e064643fcd2b2312fd0e337fa8c07f21a9bd6d1b7 WatchSource:0}: Error finding container 12f7be96c90451ce7f1f6a3e064643fcd2b2312fd0e337fa8c07f21a9bd6d1b7: Status 404 returned error can't find the container with id 12f7be96c90451ce7f1f6a3e064643fcd2b2312fd0e337fa8c07f21a9bd6d1b7 Nov 03 20:10:54.444884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:54.444849 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" event={"ID":"ebb4dcd0-d282-43ba-919b-ccd559228fa9","Type":"ContainerStarted","Data":"12f7be96c90451ce7f1f6a3e064643fcd2b2312fd0e337fa8c07f21a9bd6d1b7"} Nov 03 20:10:54.935608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:54.934877 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq"] Nov 03 20:10:54.956591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:54.954835 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq"] Nov 03 20:10:54.956591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:54.954951 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:54.960976 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:54.960947 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-433102-openshift-default-dockercfg-wjt54\"" Nov 03 20:10:55.098223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.071104 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-workload-certs\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.098223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.071189 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-envoy\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.098223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.071295 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-data\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.098223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.071373 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-credential-socket\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.098223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.071437 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-token\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.098223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.071472 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzswn\" (UniqueName: \"kubernetes.io/projected/90eb063f-fb7a-4c71-b959-8504f519eb4e-kube-api-access-tzswn\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.098223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.071594 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-workload-socket\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.098223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.071720 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90eb063f-fb7a-4c71-b959-8504f519eb4e-istiod-ca-cert\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.098223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.071772 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-podinfo\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.187602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.177044 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-workload-certs\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.187602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.177123 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-envoy\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.187602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.177217 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-data\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.187602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.177292 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-credential-socket\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.187602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.177352 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-token\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.187602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.177384 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzswn\" (UniqueName: \"kubernetes.io/projected/90eb063f-fb7a-4c71-b959-8504f519eb4e-kube-api-access-tzswn\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.187602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.177479 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-workload-socket\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.187602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.177613 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90eb063f-fb7a-4c71-b959-8504f519eb4e-istiod-ca-cert\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.187602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.177662 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-podinfo\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.187602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.187024 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-credential-socket\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.187602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.187308 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-workload-certs\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.208554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.204019 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-data\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.214555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.209896 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-workload-socket\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.214555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.210560 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90eb063f-fb7a-4c71-b959-8504f519eb4e-istiod-ca-cert\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.221555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.221193 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzswn\" (UniqueName: \"kubernetes.io/projected/90eb063f-fb7a-4c71-b959-8504f519eb4e-kube-api-access-tzswn\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.256551 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.249866 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-podinfo\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.316553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.300318 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-token\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.316553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.300491 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-envoy\") pod \"e2e-api-a-433102-openshift-default-555967f4f5-hjlkq\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.495124 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.495035 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" event={"ID":"ebb4dcd0-d282-43ba-919b-ccd559228fa9","Type":"ContainerStarted","Data":"830ab6ffee97e1ac7d2f2bc964541e3406de7a4440eb27b671f0b2ec348c725a"} Nov 03 20:10:55.587269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.587222 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:55.907698 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:10:55.907663 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90eb063f_fb7a_4c71_b959_8504f519eb4e.slice/crio-89e22a4cf6f6092b7be4830846bf78ac4e3dea4658e49055292859a5f791c1a1 WatchSource:0}: Error finding container 89e22a4cf6f6092b7be4830846bf78ac4e3dea4658e49055292859a5f791c1a1: Status 404 returned error can't find the container with id 89e22a4cf6f6092b7be4830846bf78ac4e3dea4658e49055292859a5f791c1a1 Nov 03 20:10:55.908667 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.908636 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq"] Nov 03 20:10:55.912759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.912724 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:55.912870 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.912790 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:55.912870 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:55.912818 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:10:56.515442 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:56.515403 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" event={"ID":"90eb063f-fb7a-4c71-b959-8504f519eb4e","Type":"ContainerStarted","Data":"c19adc674bea25870bbbba24bd3861386162d5ade016292a35c341536bc274c0"} Nov 03 20:10:56.515996 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:56.515470 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" event={"ID":"90eb063f-fb7a-4c71-b959-8504f519eb4e","Type":"ContainerStarted","Data":"89e22a4cf6f6092b7be4830846bf78ac4e3dea4658e49055292859a5f791c1a1"} Nov 03 20:10:56.519284 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:56.519250 5111 generic.go:356] "Generic (PLEG): container finished" podID="ebb4dcd0-d282-43ba-919b-ccd559228fa9" containerID="830ab6ffee97e1ac7d2f2bc964541e3406de7a4440eb27b671f0b2ec348c725a" exitCode=0 Nov 03 20:10:56.519449 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:56.519326 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" event={"ID":"ebb4dcd0-d282-43ba-919b-ccd559228fa9","Type":"ContainerDied","Data":"830ab6ffee97e1ac7d2f2bc964541e3406de7a4440eb27b671f0b2ec348c725a"} Nov 03 20:10:56.545561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:56.544015 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" podStartSLOduration=2.543998032 podStartE2EDuration="2.543998032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:56.538112894 +0000 UTC m=+1886.500491334" watchObservedRunningTime="2026-11-03 20:10:56.543998032 +0000 UTC m=+1886.506376471" Nov 03 20:10:56.587466 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:56.587437 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:57.142184 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:57.141806 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" Nov 03 20:10:57.196919 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:57.196865 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebb4dcd0-d282-43ba-919b-ccd559228fa9-kube-api-access-lcn69\" (UniqueName: \"kubernetes.io/projected/ebb4dcd0-d282-43ba-919b-ccd559228fa9-kube-api-access-lcn69\") pod \"ebb4dcd0-d282-43ba-919b-ccd559228fa9\" (UID: \"ebb4dcd0-d282-43ba-919b-ccd559228fa9\") " Nov 03 20:10:57.197579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:57.197548 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebb4dcd0-d282-43ba-919b-ccd559228fa9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb4dcd0-d282-43ba-919b-ccd559228fa9-maas-api-openshift-ca\") pod \"ebb4dcd0-d282-43ba-919b-ccd559228fa9\" (UID: \"ebb4dcd0-d282-43ba-919b-ccd559228fa9\") " Nov 03 20:10:57.197876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:57.197858 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebb4dcd0-d282-43ba-919b-ccd559228fa9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb4dcd0-d282-43ba-919b-ccd559228fa9-maas-api-xks-ca\") pod \"ebb4dcd0-d282-43ba-919b-ccd559228fa9\" (UID: \"ebb4dcd0-d282-43ba-919b-ccd559228fa9\") " Nov 03 20:10:57.199832 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:57.199804 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb4dcd0-d282-43ba-919b-ccd559228fa9-maas-api-openshift-ca" pod "ebb4dcd0-d282-43ba-919b-ccd559228fa9" (UID: "ebb4dcd0-d282-43ba-919b-ccd559228fa9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:10:57.202863 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:57.202837 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebb4dcd0-d282-43ba-919b-ccd559228fa9-maas-api-xks-ca" pod "ebb4dcd0-d282-43ba-919b-ccd559228fa9" (UID: "ebb4dcd0-d282-43ba-919b-ccd559228fa9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:10:57.203728 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:57.203706 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb4dcd0-d282-43ba-919b-ccd559228fa9-kube-api-access-lcn69" pod "ebb4dcd0-d282-43ba-919b-ccd559228fa9" (UID: "ebb4dcd0-d282-43ba-919b-ccd559228fa9"). InnerVolumeSpecName "kube-api-access-lcn69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:10:57.301559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:57.301510 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcn69\" (UniqueName: \"kubernetes.io/projected/ebb4dcd0-d282-43ba-919b-ccd559228fa9-kube-api-access-lcn69\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:57.301559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:57.301555 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb4dcd0-d282-43ba-919b-ccd559228fa9-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:57.301788 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:57.301571 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb4dcd0-d282-43ba-919b-ccd559228fa9-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:57.371122 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:57.371085 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7"] Nov 03 20:10:57.531268 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:57.531229 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" Nov 03 20:10:57.531268 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:57.531257 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf" event={"ID":"ebb4dcd0-d282-43ba-919b-ccd559228fa9","Type":"ContainerDied","Data":"12f7be96c90451ce7f1f6a3e064643fcd2b2312fd0e337fa8c07f21a9bd6d1b7"} Nov 03 20:10:57.531882 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:57.531307 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12f7be96c90451ce7f1f6a3e064643fcd2b2312fd0e337fa8c07f21a9bd6d1b7" Nov 03 20:10:57.594172 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:57.594144 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:58.548417 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:58.548378 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:58.556522 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:58.556490 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:10:59.215173 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:59.215141 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4"] Nov 03 20:10:59.224393 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:10:59.224357 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d928b71f-ks6d4"] Nov 03 20:11:00.186576 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.186516 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfe7c70b-e981-4d26-a41e-934f655d2838" path="/var/lib/kubelet/pods/cfe7c70b-e981-4d26-a41e-934f655d2838/volumes" Nov 03 20:11:00.215857 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.215816 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd"] Nov 03 20:11:00.216731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.216715 5111 cpu_manager.go:414] "containerMap: removing container" podUID="ebb4dcd0-d282-43ba-919b-ccd559228fa9" containerName="revoke-keys" Nov 03 20:11:00.216786 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.216733 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb4dcd0-d282-43ba-919b-ccd559228fa9" containerName="revoke-keys" Nov 03 20:11:00.216918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.216908 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb4dcd0-d282-43ba-919b-ccd559228fa9" containerName="revoke-keys" Nov 03 20:11:00.221049 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.221031 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" Nov 03 20:11:00.224825 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.224797 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc"] Nov 03 20:11:00.228640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.228623 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" Nov 03 20:11:00.252001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.251979 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-433102\"" Nov 03 20:11:00.252001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.251987 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-433102-dockercfg-rwzj4\"" Nov 03 20:11:00.252237 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.252214 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd"] Nov 03 20:11:00.267995 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.267967 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc"] Nov 03 20:11:00.350554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.349869 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dfc8d1-743e-4c51-941f-c9dd6f279f76-plugins-config-volume\") pod \"payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd\" (UID: \"80dfc8d1-743e-4c51-941f-c9dd6f279f76\") " pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" Nov 03 20:11:00.350554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.350100 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlqb9\" (UniqueName: \"kubernetes.io/projected/05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f-kube-api-access-zlqb9\") pod \"payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc\" (UID: \"05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" Nov 03 20:11:00.350554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.350144 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcfnq\" (UniqueName: \"kubernetes.io/projected/80dfc8d1-743e-4c51-941f-c9dd6f279f76-kube-api-access-fcfnq\") pod \"payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd\" (UID: \"80dfc8d1-743e-4c51-941f-c9dd6f279f76\") " pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" Nov 03 20:11:00.350554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.350249 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc\" (UID: \"05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" Nov 03 20:11:00.567625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.565313 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dfc8d1-743e-4c51-941f-c9dd6f279f76-plugins-config-volume\") pod \"payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd\" (UID: \"80dfc8d1-743e-4c51-941f-c9dd6f279f76\") " pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" Nov 03 20:11:00.789568 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.787748 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqb9\" (UniqueName: \"kubernetes.io/projected/05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f-kube-api-access-zlqb9\") pod \"payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc\" (UID: \"05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" Nov 03 20:11:00.789568 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.787847 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcfnq\" (UniqueName: \"kubernetes.io/projected/80dfc8d1-743e-4c51-941f-c9dd6f279f76-kube-api-access-fcfnq\") pod \"payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd\" (UID: \"80dfc8d1-743e-4c51-941f-c9dd6f279f76\") " pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" Nov 03 20:11:00.789568 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.787993 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc\" (UID: \"05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" Nov 03 20:11:00.792615 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.790041 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc\" (UID: \"05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" Nov 03 20:11:00.804121 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.793040 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dfc8d1-743e-4c51-941f-c9dd6f279f76-plugins-config-volume\") pod \"payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd\" (UID: \"80dfc8d1-743e-4c51-941f-c9dd6f279f76\") " pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" Nov 03 20:11:00.811761 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.806762 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcfnq\" (UniqueName: \"kubernetes.io/projected/80dfc8d1-743e-4c51-941f-c9dd6f279f76-kube-api-access-fcfnq\") pod \"payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd\" (UID: \"80dfc8d1-743e-4c51-941f-c9dd6f279f76\") " pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" Nov 03 20:11:00.812973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.812939 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqb9\" (UniqueName: \"kubernetes.io/projected/05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f-kube-api-access-zlqb9\") pod \"payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc\" (UID: \"05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" Nov 03 20:11:00.836924 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.836850 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" Nov 03 20:11:00.847259 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:00.847178 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" Nov 03 20:11:01.178346 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:11:01.178308 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80dfc8d1_743e_4c51_941f_c9dd6f279f76.slice/crio-8585aa345933e28d0e46c903bbe6589a5fb413028805682165f7efa704b074c4 WatchSource:0}: Error finding container 8585aa345933e28d0e46c903bbe6589a5fb413028805682165f7efa704b074c4: Status 404 returned error can't find the container with id 8585aa345933e28d0e46c903bbe6589a5fb413028805682165f7efa704b074c4 Nov 03 20:11:01.179502 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:01.179470 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd"] Nov 03 20:11:01.181353 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:11:01.181323 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e9b1fb_1d9a_4a20_b79b_5236d1ffde3f.slice/crio-6e88cd19e378e41392749d720b53395d92f0a0acd85124f95c6158a902c0ff6d WatchSource:0}: Error finding container 6e88cd19e378e41392749d720b53395d92f0a0acd85124f95c6158a902c0ff6d: Status 404 returned error can't find the container with id 6e88cd19e378e41392749d720b53395d92f0a0acd85124f95c6158a902c0ff6d Nov 03 20:11:01.186742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:01.186708 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc"] Nov 03 20:11:01.826553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:01.826499 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" event={"ID":"05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f","Type":"ContainerStarted","Data":"00aac47e0ff690d70bc4c53d724c7e38916bc0f43db94250cdce94211abe64ac"} Nov 03 20:11:01.826770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:01.826575 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" event={"ID":"05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f","Type":"ContainerStarted","Data":"6e88cd19e378e41392749d720b53395d92f0a0acd85124f95c6158a902c0ff6d"} Nov 03 20:11:01.826770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:01.826606 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" Nov 03 20:11:01.828835 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:01.828806 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" event={"ID":"80dfc8d1-743e-4c51-941f-c9dd6f279f76","Type":"ContainerStarted","Data":"b1479e3e56437f506354575edcc917a7b620305883779383451fea54dee74dfa"} Nov 03 20:11:01.828991 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:01.828859 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" event={"ID":"80dfc8d1-743e-4c51-941f-c9dd6f279f76","Type":"ContainerStarted","Data":"8585aa345933e28d0e46c903bbe6589a5fb413028805682165f7efa704b074c4"} Nov 03 20:11:01.829080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:01.829014 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" Nov 03 20:11:01.848612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:01.848544 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" podStartSLOduration=1.848513306 podStartE2EDuration="1.848513306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:01.847439032 +0000 UTC m=+1891.809817473" watchObservedRunningTime="2026-11-03 20:11:01.848513306 +0000 UTC m=+1891.810891745" Nov 03 20:11:01.868675 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:01.868631 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" podStartSLOduration=1.868619207 podStartE2EDuration="1.868619207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:01.862772408 +0000 UTC m=+1891.825150849" watchObservedRunningTime="2026-11-03 20:11:01.868619207 +0000 UTC m=+1891.830997644" Nov 03 20:11:02.074555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.071004 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c"] Nov 03 20:11:02.100954 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.100822 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp"] Nov 03 20:11:02.108528 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.108396 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe55076-8cz8c"] Nov 03 20:11:02.117389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.117347 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f360aa4f-l7vwp"] Nov 03 20:11:02.186290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.186255 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="245f0880-8769-447a-964a-0742bea73eb9" path="/var/lib/kubelet/pods/245f0880-8769-447a-964a-0742bea73eb9/volumes" Nov 03 20:11:02.186936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.186919 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65482e2a-7eb1-4cd8-89fa-af7c7ad6c261" path="/var/lib/kubelet/pods/65482e2a-7eb1-4cd8-89fa-af7c7ad6c261/volumes" Nov 03 20:11:02.371690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.371649 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" podUID="f2064f1d-c7ba-4dd0-9dbe-e71e4a120956" containerName="payload-processing" containerID="cri-o://dca93f6712d3b50fc96fa4928b03f37923abf68a2f02a5dd216ef3cd72dabdc9" gracePeriod=25 Nov 03 20:11:02.735860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.735809 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" Nov 03 20:11:02.842800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.842766 5111 generic.go:356] "Generic (PLEG): container finished" podID="f2064f1d-c7ba-4dd0-9dbe-e71e4a120956" containerID="dca93f6712d3b50fc96fa4928b03f37923abf68a2f02a5dd216ef3cd72dabdc9" exitCode=0 Nov 03 20:11:02.842959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.842824 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" Nov 03 20:11:02.842959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.842850 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" event={"ID":"f2064f1d-c7ba-4dd0-9dbe-e71e4a120956","Type":"ContainerDied","Data":"dca93f6712d3b50fc96fa4928b03f37923abf68a2f02a5dd216ef3cd72dabdc9"} Nov 03 20:11:02.842959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.842918 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7" event={"ID":"f2064f1d-c7ba-4dd0-9dbe-e71e4a120956","Type":"ContainerDied","Data":"4644686cd82cff97bd3fac3cccdeff9212f4ac7f6f37abecefc8dd9803d92a5e"} Nov 03 20:11:02.842959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.842946 5111 scope.go:122] "RemoveContainer" containerID="dca93f6712d3b50fc96fa4928b03f37923abf68a2f02a5dd216ef3cd72dabdc9" Nov 03 20:11:02.861037 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.860567 5111 scope.go:122] "RemoveContainer" containerID="dca93f6712d3b50fc96fa4928b03f37923abf68a2f02a5dd216ef3cd72dabdc9" Nov 03 20:11:02.861037 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:11:02.861014 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca93f6712d3b50fc96fa4928b03f37923abf68a2f02a5dd216ef3cd72dabdc9\": container with ID starting with dca93f6712d3b50fc96fa4928b03f37923abf68a2f02a5dd216ef3cd72dabdc9 not found: ID does not exist" containerID="dca93f6712d3b50fc96fa4928b03f37923abf68a2f02a5dd216ef3cd72dabdc9" Nov 03 20:11:02.861230 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.861048 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca93f6712d3b50fc96fa4928b03f37923abf68a2f02a5dd216ef3cd72dabdc9"} err="failed to get container status \"dca93f6712d3b50fc96fa4928b03f37923abf68a2f02a5dd216ef3cd72dabdc9\": rpc error: code = NotFound desc = could not find container \"dca93f6712d3b50fc96fa4928b03f37923abf68a2f02a5dd216ef3cd72dabdc9\": container with ID starting with dca93f6712d3b50fc96fa4928b03f37923abf68a2f02a5dd216ef3cd72dabdc9 not found: ID does not exist" Nov 03 20:11:02.871997 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.871976 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2064f1d-c7ba-4dd0-9dbe-e71e4a120956-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2064f1d-c7ba-4dd0-9dbe-e71e4a120956-plugins-config-volume\") pod \"f2064f1d-c7ba-4dd0-9dbe-e71e4a120956\" (UID: \"f2064f1d-c7ba-4dd0-9dbe-e71e4a120956\") " Nov 03 20:11:02.872114 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.872061 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2064f1d-c7ba-4dd0-9dbe-e71e4a120956-kube-api-access-vhmqv\" (UniqueName: \"kubernetes.io/projected/f2064f1d-c7ba-4dd0-9dbe-e71e4a120956-kube-api-access-vhmqv\") pod \"f2064f1d-c7ba-4dd0-9dbe-e71e4a120956\" (UID: \"f2064f1d-c7ba-4dd0-9dbe-e71e4a120956\") " Nov 03 20:11:02.872336 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.872309 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2064f1d-c7ba-4dd0-9dbe-e71e4a120956-plugins-config-volume" pod "f2064f1d-c7ba-4dd0-9dbe-e71e4a120956" (UID: "f2064f1d-c7ba-4dd0-9dbe-e71e4a120956"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:11:02.872809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.872780 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2064f1d-c7ba-4dd0-9dbe-e71e4a120956-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:02.874223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.874197 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2064f1d-c7ba-4dd0-9dbe-e71e4a120956-kube-api-access-vhmqv" pod "f2064f1d-c7ba-4dd0-9dbe-e71e4a120956" (UID: "f2064f1d-c7ba-4dd0-9dbe-e71e4a120956"). InnerVolumeSpecName "kube-api-access-vhmqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:02.974790 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:02.974703 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhmqv\" (UniqueName: \"kubernetes.io/projected/f2064f1d-c7ba-4dd0-9dbe-e71e4a120956-kube-api-access-vhmqv\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:03.171881 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:03.171847 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7"] Nov 03 20:11:03.178796 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:03.178772 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1962f9-77fcbfdbf6-vkqw7"] Nov 03 20:11:03.902554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:03.901645 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd"] Nov 03 20:11:04.741659 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:04.734932 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2064f1d-c7ba-4dd0-9dbe-e71e4a120956" path="/var/lib/kubelet/pods/f2064f1d-c7ba-4dd0-9dbe-e71e4a120956/volumes" Nov 03 20:11:04.780733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:04.780694 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm"] Nov 03 20:11:04.788136 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:04.788101 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-pggcm"] Nov 03 20:11:05.120104 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:11:05.120048 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:11:06.025649 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:06.025621 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb"] Nov 03 20:11:06.032553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:06.032513 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a121a291-jpjcb"] Nov 03 20:11:06.185752 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:06.185721 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a21abc9-e2e3-4d49-917f-0277dfe0bcd8" path="/var/lib/kubelet/pods/9a21abc9-e2e3-4d49-917f-0277dfe0bcd8/volumes" Nov 03 20:11:06.186220 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:06.186204 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d40208cf-757c-4b13-a2f6-722e534b1f63" path="/var/lib/kubelet/pods/d40208cf-757c-4b13-a2f6-722e534b1f63/volumes" Nov 03 20:11:08.063389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:08.062713 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz"] Nov 03 20:11:08.072681 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:08.072626 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20b93cf5-r4kvz"] Nov 03 20:11:08.186464 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:08.186429 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60956010-1d2b-4873-bff8-384c42f66e4d" path="/var/lib/kubelet/pods/60956010-1d2b-4873-bff8-384c42f66e4d/volumes" Nov 03 20:11:08.901998 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:08.901954 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" podUID="aeab908d-cb84-4a21-8caf-7b7434c3b0e8" containerName="payload-processing" containerID="cri-o://02624675b1794c5ab4351316170d302442e36efbce63b1d3bb01a7b8b4bc6267" gracePeriod=25 Nov 03 20:11:09.255150 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.255123 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" Nov 03 20:11:09.376908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.376866 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeab908d-cb84-4a21-8caf-7b7434c3b0e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeab908d-cb84-4a21-8caf-7b7434c3b0e8-plugins-config-volume\") pod \"aeab908d-cb84-4a21-8caf-7b7434c3b0e8\" (UID: \"aeab908d-cb84-4a21-8caf-7b7434c3b0e8\") " Nov 03 20:11:09.377109 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.376920 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeab908d-cb84-4a21-8caf-7b7434c3b0e8-kube-api-access-g2zqt\" (UniqueName: \"kubernetes.io/projected/aeab908d-cb84-4a21-8caf-7b7434c3b0e8-kube-api-access-g2zqt\") pod \"aeab908d-cb84-4a21-8caf-7b7434c3b0e8\" (UID: \"aeab908d-cb84-4a21-8caf-7b7434c3b0e8\") " Nov 03 20:11:09.377349 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.377322 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeab908d-cb84-4a21-8caf-7b7434c3b0e8-plugins-config-volume" pod "aeab908d-cb84-4a21-8caf-7b7434c3b0e8" (UID: "aeab908d-cb84-4a21-8caf-7b7434c3b0e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:11:09.377945 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.377922 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeab908d-cb84-4a21-8caf-7b7434c3b0e8-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:09.379336 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.379311 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeab908d-cb84-4a21-8caf-7b7434c3b0e8-kube-api-access-g2zqt" pod "aeab908d-cb84-4a21-8caf-7b7434c3b0e8" (UID: "aeab908d-cb84-4a21-8caf-7b7434c3b0e8"). InnerVolumeSpecName "kube-api-access-g2zqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:09.479433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.479365 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2zqt\" (UniqueName: \"kubernetes.io/projected/aeab908d-cb84-4a21-8caf-7b7434c3b0e8-kube-api-access-g2zqt\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:09.871074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.871040 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws"] Nov 03 20:11:09.871910 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.871895 5111 cpu_manager.go:414] "containerMap: removing container" podUID="aeab908d-cb84-4a21-8caf-7b7434c3b0e8" containerName="payload-processing" Nov 03 20:11:09.871975 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.871911 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeab908d-cb84-4a21-8caf-7b7434c3b0e8" containerName="payload-processing" Nov 03 20:11:09.871975 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.871946 5111 cpu_manager.go:414] "containerMap: removing container" podUID="f2064f1d-c7ba-4dd0-9dbe-e71e4a120956" containerName="payload-processing" Nov 03 20:11:09.871975 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.871952 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2064f1d-c7ba-4dd0-9dbe-e71e4a120956" containerName="payload-processing" Nov 03 20:11:09.872129 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.872118 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2064f1d-c7ba-4dd0-9dbe-e71e4a120956" containerName="payload-processing" Nov 03 20:11:09.872164 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.872141 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeab908d-cb84-4a21-8caf-7b7434c3b0e8" containerName="payload-processing" Nov 03 20:11:09.876430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.876405 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" Nov 03 20:11:09.882016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.881991 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-49fb2b\"" Nov 03 20:11:09.883026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.883005 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-49fb2b-dockercfg-2h5wl\"" Nov 03 20:11:09.883277 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.883250 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws"] Nov 03 20:11:09.920146 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.920114 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t"] Nov 03 20:11:09.925049 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.925029 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" Nov 03 20:11:09.940744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:09.940714 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t"] Nov 03 20:11:10.004555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.004185 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ef42c1-3bd0-4679-809b-72d16fe33a2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws\" (UID: \"e6ef42c1-3bd0-4679-809b-72d16fe33a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" Nov 03 20:11:10.004759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.004625 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmwhc\" (UniqueName: \"kubernetes.io/projected/e6ef42c1-3bd0-4679-809b-72d16fe33a2f-kube-api-access-pmwhc\") pod \"payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws\" (UID: \"e6ef42c1-3bd0-4679-809b-72d16fe33a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" Nov 03 20:11:10.118646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.108334 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t\" (UID: \"ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" Nov 03 20:11:10.118646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.108412 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v8r2\" (UniqueName: \"kubernetes.io/projected/ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7-kube-api-access-7v8r2\") pod \"payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t\" (UID: \"ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" Nov 03 20:11:10.118646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.108631 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwhc\" (UniqueName: \"kubernetes.io/projected/e6ef42c1-3bd0-4679-809b-72d16fe33a2f-kube-api-access-pmwhc\") pod \"payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws\" (UID: \"e6ef42c1-3bd0-4679-809b-72d16fe33a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" Nov 03 20:11:10.118646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.108760 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ef42c1-3bd0-4679-809b-72d16fe33a2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws\" (UID: \"e6ef42c1-3bd0-4679-809b-72d16fe33a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" Nov 03 20:11:10.118646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.109770 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ef42c1-3bd0-4679-809b-72d16fe33a2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws\" (UID: \"e6ef42c1-3bd0-4679-809b-72d16fe33a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" Nov 03 20:11:10.216607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.214249 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t\" (UID: \"ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" Nov 03 20:11:10.216607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.214324 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v8r2\" (UniqueName: \"kubernetes.io/projected/ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7-kube-api-access-7v8r2\") pod \"payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t\" (UID: \"ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" Nov 03 20:11:10.216607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.215918 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t\" (UID: \"ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" Nov 03 20:11:10.233564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.222486 5111 generic.go:356] "Generic (PLEG): container finished" podID="aeab908d-cb84-4a21-8caf-7b7434c3b0e8" containerID="02624675b1794c5ab4351316170d302442e36efbce63b1d3bb01a7b8b4bc6267" exitCode=0 Nov 03 20:11:10.243555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.243267 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmwhc\" (UniqueName: \"kubernetes.io/projected/e6ef42c1-3bd0-4679-809b-72d16fe33a2f-kube-api-access-pmwhc\") pod \"payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws\" (UID: \"e6ef42c1-3bd0-4679-809b-72d16fe33a2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" Nov 03 20:11:10.262135 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.262055 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v8r2\" (UniqueName: \"kubernetes.io/projected/ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7-kube-api-access-7v8r2\") pod \"payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t\" (UID: \"ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" Nov 03 20:11:10.269699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.269670 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" Nov 03 20:11:10.391198 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.391042 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" Nov 03 20:11:10.394308 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.393422 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" event={"ID":"aeab908d-cb84-4a21-8caf-7b7434c3b0e8","Type":"ContainerDied","Data":"02624675b1794c5ab4351316170d302442e36efbce63b1d3bb01a7b8b4bc6267"} Nov 03 20:11:10.394308 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.393479 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd" event={"ID":"aeab908d-cb84-4a21-8caf-7b7434c3b0e8","Type":"ContainerDied","Data":"1e5ef135d569ed8cd0ac1fe1bdbc1f49d0eeab74d978aa49ccd1a7095e770c49"} Nov 03 20:11:10.394308 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.393508 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq"] Nov 03 20:11:10.394308 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.393557 5111 scope.go:122] "RemoveContainer" containerID="02624675b1794c5ab4351316170d302442e36efbce63b1d3bb01a7b8b4bc6267" Nov 03 20:11:10.402331 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.402299 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51249b31-77vgq"] Nov 03 20:11:10.463430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.461688 5111 scope.go:122] "RemoveContainer" containerID="02624675b1794c5ab4351316170d302442e36efbce63b1d3bb01a7b8b4bc6267" Nov 03 20:11:10.463430 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:11:10.462289 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02624675b1794c5ab4351316170d302442e36efbce63b1d3bb01a7b8b4bc6267\": container with ID starting with 02624675b1794c5ab4351316170d302442e36efbce63b1d3bb01a7b8b4bc6267 not found: ID does not exist" containerID="02624675b1794c5ab4351316170d302442e36efbce63b1d3bb01a7b8b4bc6267" Nov 03 20:11:10.463430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.462335 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02624675b1794c5ab4351316170d302442e36efbce63b1d3bb01a7b8b4bc6267"} err="failed to get container status \"02624675b1794c5ab4351316170d302442e36efbce63b1d3bb01a7b8b4bc6267\": rpc error: code = NotFound desc = could not find container \"02624675b1794c5ab4351316170d302442e36efbce63b1d3bb01a7b8b4bc6267\": container with ID starting with 02624675b1794c5ab4351316170d302442e36efbce63b1d3bb01a7b8b4bc6267 not found: ID does not exist" Nov 03 20:11:10.467108 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.466756 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd"] Nov 03 20:11:10.479521 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.479487 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9dc48-9d6cf9dd8-545xd"] Nov 03 20:11:10.525738 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.524849 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" Nov 03 20:11:10.730564 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:11:10.730513 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ef42c1_3bd0_4679_809b_72d16fe33a2f.slice/crio-1b7e6df9cad0865b3d0e26168fcea7d056631c96737c11c00d75879462edaf21 WatchSource:0}: Error finding container 1b7e6df9cad0865b3d0e26168fcea7d056631c96737c11c00d75879462edaf21: Status 404 returned error can't find the container with id 1b7e6df9cad0865b3d0e26168fcea7d056631c96737c11c00d75879462edaf21 Nov 03 20:11:10.731285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.731257 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws"] Nov 03 20:11:10.862999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:10.862965 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t"] Nov 03 20:11:10.863120 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:11:10.862969 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccef9d55_c21a_48c9_ad75_fc2e1d5e69c7.slice/crio-7f8780137ecafcfee4b9ea1fb5eaaddc8b2b81c3a3c41c0ddb3fd0fed8a49da8 WatchSource:0}: Error finding container 7f8780137ecafcfee4b9ea1fb5eaaddc8b2b81c3a3c41c0ddb3fd0fed8a49da8: Status 404 returned error can't find the container with id 7f8780137ecafcfee4b9ea1fb5eaaddc8b2b81c3a3c41c0ddb3fd0fed8a49da8 Nov 03 20:11:11.033349 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:11.033314 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6"] Nov 03 20:11:11.043719 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:11.042226 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1028db-g9ch6"] Nov 03 20:11:11.259847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:11.259736 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" event={"ID":"ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7","Type":"ContainerStarted","Data":"73baac09f3730e7252efed2bea1887ab5da8bb2ce6608f87997db9ddc894e8a3"} Nov 03 20:11:11.259847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:11.259810 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" event={"ID":"ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7","Type":"ContainerStarted","Data":"7f8780137ecafcfee4b9ea1fb5eaaddc8b2b81c3a3c41c0ddb3fd0fed8a49da8"} Nov 03 20:11:11.260101 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:11.259871 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" Nov 03 20:11:11.261922 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:11.261891 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" event={"ID":"e6ef42c1-3bd0-4679-809b-72d16fe33a2f","Type":"ContainerStarted","Data":"f3a1d2c04b941c6ce0ba897066682c01990a1315ec615745b290b8b39c4dcfe4"} Nov 03 20:11:11.262066 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:11.261930 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" event={"ID":"e6ef42c1-3bd0-4679-809b-72d16fe33a2f","Type":"ContainerStarted","Data":"1b7e6df9cad0865b3d0e26168fcea7d056631c96737c11c00d75879462edaf21"} Nov 03 20:11:11.262066 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:11.261974 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" Nov 03 20:11:11.277458 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:11.277398 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" podStartSLOduration=2.277384154 podStartE2EDuration="2.277384154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:11.275286293 +0000 UTC m=+1901.237664733" watchObservedRunningTime="2026-11-03 20:11:11.277384154 +0000 UTC m=+1901.239762591" Nov 03 20:11:11.297612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:11.297561 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" podStartSLOduration=2.297527063 podStartE2EDuration="2.297527063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:11.295149027 +0000 UTC m=+1901.257527473" watchObservedRunningTime="2026-11-03 20:11:11.297527063 +0000 UTC m=+1901.259905497" Nov 03 20:11:12.210084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.210045 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2050bea5-d734-487f-ba88-5eb6b4d67f81" path="/var/lib/kubelet/pods/2050bea5-d734-487f-ba88-5eb6b4d67f81/volumes" Nov 03 20:11:12.211248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.211220 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1849d3-b100-41a1-b92a-df99efc5657f" path="/var/lib/kubelet/pods/4b1849d3-b100-41a1-b92a-df99efc5657f/volumes" Nov 03 20:11:12.212093 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.212065 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeab908d-cb84-4a21-8caf-7b7434c3b0e8" path="/var/lib/kubelet/pods/aeab908d-cb84-4a21-8caf-7b7434c3b0e8/volumes" Nov 03 20:11:12.212613 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.212592 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66"] Nov 03 20:11:12.221041 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.221015 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" Nov 03 20:11:12.226114 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.226085 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66"] Nov 03 20:11:12.226994 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.226968 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-acd98d21-dockercfg-nwmnt\"" Nov 03 20:11:12.248722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.248685 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-acd98d21\"" Nov 03 20:11:12.412552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.401505 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldrct\" (UniqueName: \"kubernetes.io/projected/0e61e6c5-045f-43e5-a944-a7f3aa50522b-kube-api-access-ldrct\") pod \"payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66\" (UID: \"0e61e6c5-045f-43e5-a944-a7f3aa50522b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" Nov 03 20:11:12.412552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.401654 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e61e6c5-045f-43e5-a944-a7f3aa50522b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66\" (UID: \"0e61e6c5-045f-43e5-a944-a7f3aa50522b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" Nov 03 20:11:12.509585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.505050 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldrct\" (UniqueName: \"kubernetes.io/projected/0e61e6c5-045f-43e5-a944-a7f3aa50522b-kube-api-access-ldrct\") pod \"payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66\" (UID: \"0e61e6c5-045f-43e5-a944-a7f3aa50522b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" Nov 03 20:11:12.509585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.505215 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e61e6c5-045f-43e5-a944-a7f3aa50522b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66\" (UID: \"0e61e6c5-045f-43e5-a944-a7f3aa50522b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" Nov 03 20:11:12.550384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.541195 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e61e6c5-045f-43e5-a944-a7f3aa50522b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66\" (UID: \"0e61e6c5-045f-43e5-a944-a7f3aa50522b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" Nov 03 20:11:12.611331 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.611297 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldrct\" (UniqueName: \"kubernetes.io/projected/0e61e6c5-045f-43e5-a944-a7f3aa50522b-kube-api-access-ldrct\") pod \"payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66\" (UID: \"0e61e6c5-045f-43e5-a944-a7f3aa50522b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" Nov 03 20:11:12.843327 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.843239 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" Nov 03 20:11:12.847158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.847110 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" Nov 03 20:11:12.847281 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:12.847261 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" Nov 03 20:11:13.136578 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:13.136412 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d"] Nov 03 20:11:13.162231 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:13.162193 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35443837-swl8d"] Nov 03 20:11:13.175868 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:11:13.175819 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e61e6c5_045f_43e5_a944_a7f3aa50522b.slice/crio-08113c86d09f966b1476f30583aa1495816a83023d4f5bec133b8f55f33be479 WatchSource:0}: Error finding container 08113c86d09f966b1476f30583aa1495816a83023d4f5bec133b8f55f33be479: Status 404 returned error can't find the container with id 08113c86d09f966b1476f30583aa1495816a83023d4f5bec133b8f55f33be479 Nov 03 20:11:13.178705 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:13.178662 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66"] Nov 03 20:11:13.405414 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:13.405317 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" event={"ID":"0e61e6c5-045f-43e5-a944-a7f3aa50522b","Type":"ContainerStarted","Data":"f690f2b6309a6d9b0804ed9b2507d7e379bb9694312225222bcc3faf44e7091f"} Nov 03 20:11:13.405414 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:13.405386 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" Nov 03 20:11:13.405414 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:13.405411 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" event={"ID":"0e61e6c5-045f-43e5-a944-a7f3aa50522b","Type":"ContainerStarted","Data":"08113c86d09f966b1476f30583aa1495816a83023d4f5bec133b8f55f33be479"} Nov 03 20:11:13.424713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:13.424641 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" podStartSLOduration=1.4246244209999999 podStartE2EDuration="1.424624421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:13.424284304 +0000 UTC m=+1903.386662743" watchObservedRunningTime="2026-11-03 20:11:13.424624421 +0000 UTC m=+1903.387002863" Nov 03 20:11:14.337107 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:14.337048 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f" path="/var/lib/kubelet/pods/6f7ac9a4-aa54-4eed-a40f-fb0d77a6ee4f/volumes" Nov 03 20:11:17.023596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:17.008334 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:11:17.701559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:17.697652 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2"] Nov 03 20:11:17.859557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:17.834650 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:17.909606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:17.863156 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-workload-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:17.914558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:17.909908 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh927\" (UniqueName: \"kubernetes.io/projected/937bc0e2-ac02-4c96-83ca-21e0c9b82506-kube-api-access-xh927\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:17.923559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:17.891891 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b9dc48-openshift-default-dockercfg-wg8jd\"" Nov 03 20:11:18.012710 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:17.871564 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz"] Nov 03 20:11:18.030560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:17.951462 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-workload-certs\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.030560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.024616 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-envoy\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.030560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.024653 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-podinfo\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.030560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.024757 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-credential-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.030560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.024792 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istiod-ca-cert\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.030560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.025106 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-data\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.030560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.025274 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-token\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.201557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.170896 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-data\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.201557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.171054 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-token\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.201557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.171156 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-workload-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.201557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.171217 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh927\" (UniqueName: \"kubernetes.io/projected/937bc0e2-ac02-4c96-83ca-21e0c9b82506-kube-api-access-xh927\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.201557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.171354 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-workload-certs\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.201557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.171398 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-envoy\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.201557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.171427 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-podinfo\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.201557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.171512 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-credential-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.201557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.171561 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istiod-ca-cert\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.201557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.181270 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-data\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.201557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.181906 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-workload-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.201557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.182395 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-workload-certs\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.231329 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.230847 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istiod-ca-cert\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.232102 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.232076 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-credential-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.360982 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.329921 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-podinfo\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.465597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.441143 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2"] Nov 03 20:11:18.465597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.441316 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz"] Nov 03 20:11:18.465597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.445351 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.561483 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.561254 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh927\" (UniqueName: \"kubernetes.io/projected/937bc0e2-ac02-4c96-83ca-21e0c9b82506-kube-api-access-xh927\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.563590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.563560 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-token\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.605860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.592453 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-envoy\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.605860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.598560 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-credential-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.605860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.598619 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7gwj\" (UniqueName: \"kubernetes.io/projected/4ebf23a0-415f-452a-a530-aee813434ba7-kube-api-access-z7gwj\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.605860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.598726 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-istio-envoy\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.605860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.598869 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebf23a0-415f-452a-a530-aee813434ba7-istio-token\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.605860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.598955 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebf23a0-415f-452a-a530-aee813434ba7-istiod-ca-cert\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.605860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.599003 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-workload-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.605860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.599129 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-workload-certs\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.605860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.599223 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-istio-data\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.605860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.599352 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebf23a0-415f-452a-a530-aee813434ba7-istio-podinfo\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.708556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.708488 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebf23a0-415f-452a-a530-aee813434ba7-istiod-ca-cert\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.708764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.708580 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-workload-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.708764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.708684 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-workload-certs\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.708764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.708756 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-istio-data\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.708959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.708843 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebf23a0-415f-452a-a530-aee813434ba7-istio-podinfo\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.732562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.709013 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-credential-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.732562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.709051 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gwj\" (UniqueName: \"kubernetes.io/projected/4ebf23a0-415f-452a-a530-aee813434ba7-kube-api-access-z7gwj\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.732562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.709119 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-istio-envoy\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.732562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.709251 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebf23a0-415f-452a-a530-aee813434ba7-istio-token\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.732562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.722480 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2"] Nov 03 20:11:18.753559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.722970 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-istio-data\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.753559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.742061 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-workload-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.753559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.742746 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebf23a0-415f-452a-a530-aee813434ba7-istiod-ca-cert\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.753559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.743028 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-credential-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.753559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.743796 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-workload-certs\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:18.892281 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.821939 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:18.892281 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.891238 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebf23a0-415f-452a-a530-aee813434ba7-istio-podinfo\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:19.073683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.910989 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:11:19.073683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.912069 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:11:19.073683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.918319 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-istio-envoy\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:19.073683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.955592 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:11:19.073683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:18.958812 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:11:19.073683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:19.043168 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:11:19.166569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:19.147427 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:11:19.235214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:19.197449 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7gwj\" (UniqueName: \"kubernetes.io/projected/4ebf23a0-415f-452a-a530-aee813434ba7-kube-api-access-z7gwj\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:19.235214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:19.210658 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:11:19.369701 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:19.369671 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43b392f6-lqqd2"] Nov 03 20:11:19.386703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:19.386664 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:11:19.471121 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:19.454703 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 20:11:19.564573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:19.564521 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebf23a0-415f-452a-a530-aee813434ba7-istio-token\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:19.601671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:19.601618 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:11:19.638743 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:19.638586 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f"] Nov 03 20:11:19.668763 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:19.668729 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bc1749b-jgq6f"] Nov 03 20:11:19.860793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:19.860757 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:19.920610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:19.916902 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:11:19.920610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:19.917442 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:11:20.020888 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.020848 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:11:20.026550 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.024258 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:11:20.123896 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.123847 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb"] Nov 03 20:11:20.125696 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.125675 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:11:20.156667 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.156632 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48d573db-qx4vb"] Nov 03 20:11:20.175352 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.175137 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:11:20.261444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.261406 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:11:20.261633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.261485 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:11:20.261633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.261527 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:11:20.273420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.273387 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e145ff2-89d8-45f2-b777-a78e17636e47" path="/var/lib/kubelet/pods/7e145ff2-89d8-45f2-b777-a78e17636e47/volumes" Nov 03 20:11:20.274168 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.274146 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e" path="/var/lib/kubelet/pods/ace0c2ed-b808-4d4c-bf6c-1910edbd3b0e/volumes" Nov 03 20:11:20.274868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.274828 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb4377fe-0214-4470-9b01-f7b42f268f27" path="/var/lib/kubelet/pods/cb4377fe-0214-4470-9b01-f7b42f268f27/volumes" Nov 03 20:11:20.275336 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.275317 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2"] Nov 03 20:11:20.602119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.601186 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" event={"ID":"937bc0e2-ac02-4c96-83ca-21e0c9b82506","Type":"ContainerStarted","Data":"b18bc6e8eb6ccc67aea97df1905623ac5f167663ab445745387f2a7651d8664d"} Nov 03 20:11:20.602119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.601252 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" event={"ID":"937bc0e2-ac02-4c96-83ca-21e0c9b82506","Type":"ContainerStarted","Data":"66d140ea1a41d0ac74196f071acca91594474f538b8e947302c2393636d91a63"} Nov 03 20:11:20.610929 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.610884 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz"] Nov 03 20:11:20.630102 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.629667 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:11:20.630102 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.629754 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:11:20.630102 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.629801 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:11:20.632840 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.632767 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" podStartSLOduration=3.632748966 podStartE2EDuration="3.632748966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:20.619279653 +0000 UTC m=+1910.581658092" watchObservedRunningTime="2026-11-03 20:11:20.632748966 +0000 UTC m=+1910.595127405" Nov 03 20:11:20.822597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:20.822562 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:21.063476 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:21.063299 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw"] Nov 03 20:11:21.070292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:21.070263 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7618a46-7bdzw"] Nov 03 20:11:21.609762 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:21.609718 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" event={"ID":"4ebf23a0-415f-452a-a530-aee813434ba7","Type":"ContainerStarted","Data":"8a09a7f74b5eccd9a234185a51f94e5a6eccc38aa230d2ccf206624f8468062b"} Nov 03 20:11:21.610195 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:21.609785 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" event={"ID":"4ebf23a0-415f-452a-a530-aee813434ba7","Type":"ContainerStarted","Data":"f926410577950d7480b67669d62450f1ebf7d30a688898010997a64169b398d1"} Nov 03 20:11:21.633947 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:21.633897 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" podStartSLOduration=4.633881945 podStartE2EDuration="4.633881945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:21.629248674 +0000 UTC m=+1911.591627113" watchObservedRunningTime="2026-11-03 20:11:21.633881945 +0000 UTC m=+1911.596260383" Nov 03 20:11:21.824724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:21.824694 5111 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 03 20:11:21.824924 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:21.824751 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" podUID="937bc0e2-ac02-4c96-83ca-21e0c9b82506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 03 20:11:21.872553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:21.871591 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:21.896558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:21.888659 5111 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 20:11:21.896558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:21.888727 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" podUID="4ebf23a0-415f-452a-a530-aee813434ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 20:11:22.217416 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:22.217339 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="615f29b1-a11e-4b5d-a89b-dfefb9b91c28" path="/var/lib/kubelet/pods/615f29b1-a11e-4b5d-a89b-dfefb9b91c28/volumes" Nov 03 20:11:22.398235 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:22.398041 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" Nov 03 20:11:22.398719 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:22.398693 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" Nov 03 20:11:22.824747 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:22.824175 5111 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 03 20:11:22.824747 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:22.824237 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" podUID="937bc0e2-ac02-4c96-83ca-21e0c9b82506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 03 20:11:22.879559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:22.873871 5111 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 20:11:22.879559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:22.873961 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" podUID="4ebf23a0-415f-452a-a530-aee813434ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 20:11:23.840730 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:23.840686 5111 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 03 20:11:23.841238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:23.840766 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" podUID="937bc0e2-ac02-4c96-83ca-21e0c9b82506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 03 20:11:23.927805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:23.927764 5111 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 20:11:23.928123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:23.928093 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" podUID="4ebf23a0-415f-452a-a530-aee813434ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 20:11:24.482593 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:24.482550 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" Nov 03 20:11:24.824759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:24.824669 5111 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 03 20:11:24.825006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:24.824977 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" podUID="937bc0e2-ac02-4c96-83ca-21e0c9b82506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 03 20:11:24.929588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:24.929075 5111 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 20:11:24.929588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:24.929169 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" podUID="4ebf23a0-415f-452a-a530-aee813434ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 20:11:25.872716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:25.861250 5111 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 03 20:11:25.872716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:25.861333 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" podUID="937bc0e2-ac02-4c96-83ca-21e0c9b82506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 03 20:11:25.935403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:25.935315 5111 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 20:11:25.989554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:25.963638 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" podUID="4ebf23a0-415f-452a-a530-aee813434ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 20:11:26.437601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:26.437552 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:11:26.642019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:26.641985 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7"] Nov 03 20:11:26.642256 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:26.642234 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1962f9-pxtz7"] Nov 03 20:11:26.826438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:26.824439 5111 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 03 20:11:26.826438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:26.824495 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" podUID="937bc0e2-ac02-4c96-83ca-21e0c9b82506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 03 20:11:26.885561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:26.883812 5111 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 20:11:26.885561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:26.883888 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" podUID="4ebf23a0-415f-452a-a530-aee813434ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 20:11:27.822791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:27.822736 5111 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 03 20:11:27.823296 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:27.822842 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" podUID="937bc0e2-ac02-4c96-83ca-21e0c9b82506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 03 20:11:27.877846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:27.874280 5111 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 20:11:27.877846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:27.874341 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" podUID="4ebf23a0-415f-452a-a530-aee813434ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 20:11:28.062717 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:28.060619 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f"] Nov 03 20:11:28.090616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:28.084449 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd09552-5mh2f"] Nov 03 20:11:28.196686 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:28.196635 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a7e2d25-c018-4169-9dca-165fc108463a" path="/var/lib/kubelet/pods/0a7e2d25-c018-4169-9dca-165fc108463a/volumes" Nov 03 20:11:28.197604 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:28.197575 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d50e44-d18a-4573-a9ea-c86ef7f1a227" path="/var/lib/kubelet/pods/a0d50e44-d18a-4573-a9ea-c86ef7f1a227/volumes" Nov 03 20:11:28.829345 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:28.829312 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:28.829877 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:28.829803 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:28.830401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:28.830380 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:28.875177 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:28.875089 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:28.875496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:28.875469 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:28.876444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:28.876425 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:31.036309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:31.036273 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4"] Nov 03 20:11:31.050820 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:31.050786 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65543e7a-rp6c4"] Nov 03 20:11:31.377121 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:31.373769 5111 scope.go:122] "RemoveContainer" containerID="29cb735bef6d13bdc62970ce8d59f6818b290990a9dc2d0627927af23573350a" Nov 03 20:11:31.415878 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:31.415847 5111 scope.go:122] "RemoveContainer" containerID="7f74ecfc57e2923906aaaebe53d2991d079bd444f27a996004a2fd8eb2229645" Nov 03 20:11:31.441663 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:31.441638 5111 scope.go:122] "RemoveContainer" containerID="6198d922a2e60c32d3f21bf4530b4fc983738185fa592ee45c222c6b8dc94724" Nov 03 20:11:31.453654 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:31.453631 5111 scope.go:122] "RemoveContainer" containerID="28cfe07dd097152ebfa0e067e51b02b5c0332d109f463c9f5e47e0687f224994" Nov 03 20:11:31.468629 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:31.468495 5111 scope.go:122] "RemoveContainer" containerID="4160924f0932ab43a0cb0d1eacd5528bf0aa63d5d80673ca08ee9ac14c0956fd" Nov 03 20:11:31.488416 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:31.488319 5111 scope.go:122] "RemoveContainer" containerID="05f70d46188789c08cc80e9a7aa997a4dc418d58604ded3acd5ce0e886093c26" Nov 03 20:11:31.501245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:31.501218 5111 scope.go:122] "RemoveContainer" containerID="da29161987bb6c06b2efe57549218ff9cf49daf1c922c4c9c912e3e65638191d" Nov 03 20:11:31.527632 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:31.523723 5111 scope.go:122] "RemoveContainer" containerID="de0f16263e507e8489f073c0996472ef4f5e13327d2144bd5c01050d9387383a" Nov 03 20:11:31.568282 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:31.568248 5111 scope.go:122] "RemoveContainer" containerID="e206111691da4522d2e1846fac0fe8a287a367c0c4b652ee67bcbf3bfca7f36c" Nov 03 20:11:31.595452 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:31.595428 5111 scope.go:122] "RemoveContainer" containerID="a48905df019bfbb67c2f7472bde88c8eb81b0bda3d10df3830c297059cbf1353" Nov 03 20:11:31.608163 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:31.608143 5111 scope.go:122] "RemoveContainer" containerID="61ed8d8180cbe69d50b6cbdaeba88e98ec17106f1e7e9f390d81c855e73282bd" Nov 03 20:11:32.084026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:32.082692 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c"] Nov 03 20:11:32.090264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:32.090196 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f9b761-7l27c"] Nov 03 20:11:32.190311 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:32.190274 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22218915-cc64-4ed1-8cc9-9f4aef1d9c5a" path="/var/lib/kubelet/pods/22218915-cc64-4ed1-8cc9-9f4aef1d9c5a/volumes" Nov 03 20:11:32.190996 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:32.190971 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8440d1f-afd2-4f2d-b17e-cbb699b51444" path="/var/lib/kubelet/pods/b8440d1f-afd2-4f2d-b17e-cbb699b51444/volumes" Nov 03 20:11:32.917184 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:32.917148 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d"] Nov 03 20:11:32.917660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:32.917628 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerName="istio-proxy" containerID="cri-o://f3749975f46f05c57663e40bcd88039b67509c020b4e6748c4eb7b558075e9dd" gracePeriod=30 Nov 03 20:11:33.389609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:33.387956 5111 patch_prober.go:29] interesting pod/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:11:33.389609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:33.388027 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:11:35.069755 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:11:35.069709 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:11:38.315291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.315265 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:11:38.400128 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.400093 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-envoy\") pod \"17938ed6-2f40-4c8a-84f4-97bda29142cc\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " Nov 03 20:11:38.400385 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.400368 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-data\") pod \"17938ed6-2f40-4c8a-84f4-97bda29142cc\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " Nov 03 20:11:38.400498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.400407 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-workload-certs\") pod \"17938ed6-2f40-4c8a-84f4-97bda29142cc\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " Nov 03 20:11:38.400498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.400470 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17938ed6-2f40-4c8a-84f4-97bda29142cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17938ed6-2f40-4c8a-84f4-97bda29142cc-istiod-ca-cert\") pod \"17938ed6-2f40-4c8a-84f4-97bda29142cc\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " Nov 03 20:11:38.400698 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.400550 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-credential-socket\") pod \"17938ed6-2f40-4c8a-84f4-97bda29142cc\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " Nov 03 20:11:38.400698 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.400620 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-podinfo\") pod \"17938ed6-2f40-4c8a-84f4-97bda29142cc\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " Nov 03 20:11:38.400809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.400728 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-token\" (UniqueName: \"kubernetes.io/projected/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-token\") pod \"17938ed6-2f40-4c8a-84f4-97bda29142cc\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " Nov 03 20:11:38.400872 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.400806 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-workload-certs" pod "17938ed6-2f40-4c8a-84f4-97bda29142cc" (UID: "17938ed6-2f40-4c8a-84f4-97bda29142cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:38.400872 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.400815 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-workload-socket\") pod \"17938ed6-2f40-4c8a-84f4-97bda29142cc\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " Nov 03 20:11:38.400977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.400922 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17938ed6-2f40-4c8a-84f4-97bda29142cc-kube-api-access-n5vvs\" (UniqueName: \"kubernetes.io/projected/17938ed6-2f40-4c8a-84f4-97bda29142cc-kube-api-access-n5vvs\") pod \"17938ed6-2f40-4c8a-84f4-97bda29142cc\" (UID: \"17938ed6-2f40-4c8a-84f4-97bda29142cc\") " Nov 03 20:11:38.401047 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.401021 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-workload-socket" pod "17938ed6-2f40-4c8a-84f4-97bda29142cc" (UID: "17938ed6-2f40-4c8a-84f4-97bda29142cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:38.401312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.401288 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-data" pod "17938ed6-2f40-4c8a-84f4-97bda29142cc" (UID: "17938ed6-2f40-4c8a-84f4-97bda29142cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:38.401434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.401353 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17938ed6-2f40-4c8a-84f4-97bda29142cc-istiod-ca-cert" pod "17938ed6-2f40-4c8a-84f4-97bda29142cc" (UID: "17938ed6-2f40-4c8a-84f4-97bda29142cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:11:38.401641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.401619 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-credential-socket" pod "17938ed6-2f40-4c8a-84f4-97bda29142cc" (UID: "17938ed6-2f40-4c8a-84f4-97bda29142cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:38.402347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.402325 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:38.402471 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.402457 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:38.402640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.402623 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17938ed6-2f40-4c8a-84f4-97bda29142cc-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:38.402754 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.402741 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:38.402849 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.402837 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:38.403315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.403296 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-envoy" pod "17938ed6-2f40-4c8a-84f4-97bda29142cc" (UID: "17938ed6-2f40-4c8a-84f4-97bda29142cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:38.404103 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.404077 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17938ed6-2f40-4c8a-84f4-97bda29142cc-kube-api-access-n5vvs" pod "17938ed6-2f40-4c8a-84f4-97bda29142cc" (UID: "17938ed6-2f40-4c8a-84f4-97bda29142cc"). InnerVolumeSpecName "kube-api-access-n5vvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:38.404596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.404569 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-podinfo" pod "17938ed6-2f40-4c8a-84f4-97bda29142cc" (UID: "17938ed6-2f40-4c8a-84f4-97bda29142cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:11:38.405148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.405119 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-token" pod "17938ed6-2f40-4c8a-84f4-97bda29142cc" (UID: "17938ed6-2f40-4c8a-84f4-97bda29142cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:38.505659 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.505618 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:38.505659 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.505659 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5vvs\" (UniqueName: \"kubernetes.io/projected/17938ed6-2f40-4c8a-84f4-97bda29142cc-kube-api-access-n5vvs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:38.505916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.505676 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:38.505916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.505690 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17938ed6-2f40-4c8a-84f4-97bda29142cc-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:38.631713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.631679 5111 generic.go:356] "Generic (PLEG): container finished" podID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerID="f3749975f46f05c57663e40bcd88039b67509c020b4e6748c4eb7b558075e9dd" exitCode=0 Nov 03 20:11:38.631890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.631729 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" Nov 03 20:11:38.631890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.631761 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" event={"ID":"17938ed6-2f40-4c8a-84f4-97bda29142cc","Type":"ContainerDied","Data":"f3749975f46f05c57663e40bcd88039b67509c020b4e6748c4eb7b558075e9dd"} Nov 03 20:11:38.631890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.631832 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d" event={"ID":"17938ed6-2f40-4c8a-84f4-97bda29142cc","Type":"ContainerDied","Data":"f2e3dd3737b4da16e98db05a1943bd08e69b490308516e017f3516dda537bef1"} Nov 03 20:11:38.631890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.631852 5111 scope.go:122] "RemoveContainer" containerID="f3749975f46f05c57663e40bcd88039b67509c020b4e6748c4eb7b558075e9dd" Nov 03 20:11:38.645664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.645644 5111 scope.go:122] "RemoveContainer" containerID="aa025bd892cda3c538628a7c854d8a16a862172523c2d43a2b1ab3209f664e23" Nov 03 20:11:38.657950 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.657918 5111 scope.go:122] "RemoveContainer" containerID="f3749975f46f05c57663e40bcd88039b67509c020b4e6748c4eb7b558075e9dd" Nov 03 20:11:38.658236 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:11:38.658217 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3749975f46f05c57663e40bcd88039b67509c020b4e6748c4eb7b558075e9dd\": container with ID starting with f3749975f46f05c57663e40bcd88039b67509c020b4e6748c4eb7b558075e9dd not found: ID does not exist" containerID="f3749975f46f05c57663e40bcd88039b67509c020b4e6748c4eb7b558075e9dd" Nov 03 20:11:38.658314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.658243 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3749975f46f05c57663e40bcd88039b67509c020b4e6748c4eb7b558075e9dd"} err="failed to get container status \"f3749975f46f05c57663e40bcd88039b67509c020b4e6748c4eb7b558075e9dd\": rpc error: code = NotFound desc = could not find container \"f3749975f46f05c57663e40bcd88039b67509c020b4e6748c4eb7b558075e9dd\": container with ID starting with f3749975f46f05c57663e40bcd88039b67509c020b4e6748c4eb7b558075e9dd not found: ID does not exist" Nov 03 20:11:38.658314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.658264 5111 scope.go:122] "RemoveContainer" containerID="aa025bd892cda3c538628a7c854d8a16a862172523c2d43a2b1ab3209f664e23" Nov 03 20:11:38.658605 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:11:38.658581 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa025bd892cda3c538628a7c854d8a16a862172523c2d43a2b1ab3209f664e23\": container with ID starting with aa025bd892cda3c538628a7c854d8a16a862172523c2d43a2b1ab3209f664e23 not found: ID does not exist" containerID="aa025bd892cda3c538628a7c854d8a16a862172523c2d43a2b1ab3209f664e23" Nov 03 20:11:38.658674 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.658611 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa025bd892cda3c538628a7c854d8a16a862172523c2d43a2b1ab3209f664e23"} err="failed to get container status \"aa025bd892cda3c538628a7c854d8a16a862172523c2d43a2b1ab3209f664e23\": rpc error: code = NotFound desc = could not find container \"aa025bd892cda3c538628a7c854d8a16a862172523c2d43a2b1ab3209f664e23\": container with ID starting with aa025bd892cda3c538628a7c854d8a16a862172523c2d43a2b1ab3209f664e23 not found: ID does not exist" Nov 03 20:11:38.659920 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.659897 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d"] Nov 03 20:11:38.664914 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:38.664892 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1962f9-openshift-default-56464595f-d2j7d"] Nov 03 20:11:40.187384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.187352 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" path="/var/lib/kubelet/pods/17938ed6-2f40-4c8a-84f4-97bda29142cc/volumes" Nov 03 20:11:40.519942 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.519624 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x"] Nov 03 20:11:40.521033 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.520873 5111 cpu_manager.go:414] "containerMap: removing container" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerName="istio-proxy" Nov 03 20:11:40.521033 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.520893 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerName="istio-proxy" Nov 03 20:11:40.521033 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.520908 5111 cpu_manager.go:414] "containerMap: removing container" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerName="istio-proxy" Nov 03 20:11:40.521033 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.520918 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerName="istio-proxy" Nov 03 20:11:40.521344 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.521109 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerName="istio-proxy" Nov 03 20:11:40.521344 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.521119 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="17938ed6-2f40-4c8a-84f4-97bda29142cc" containerName="istio-proxy" Nov 03 20:11:40.527293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.527273 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.531489 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.531183 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-caa01a-openshift-default-dockercfg-hrwsz\"" Nov 03 20:11:40.556824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.556795 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x"] Nov 03 20:11:40.557406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.557377 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-workload-socket\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.557591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.557502 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istiod-ca-cert\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.557591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.557576 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-podinfo\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.557709 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.557623 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-workload-certs\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.557709 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.557650 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-data\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.557819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.557744 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-envoy\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.557819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.557810 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8wwq\" (UniqueName: \"kubernetes.io/projected/e837c3cb-1bf5-4774-a9e1-a378f67048c1-kube-api-access-j8wwq\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.557920 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.557891 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-credential-socket\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.558060 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.558033 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-token\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.669556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.664098 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istiod-ca-cert\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.669556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.664212 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-podinfo\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.669556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.664270 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-workload-certs\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.669556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.664299 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-data\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.669556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.664417 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-envoy\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.669556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.664494 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8wwq\" (UniqueName: \"kubernetes.io/projected/e837c3cb-1bf5-4774-a9e1-a378f67048c1-kube-api-access-j8wwq\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.669556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.664602 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-credential-socket\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.669556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.664769 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-token\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.669556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.664932 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-workload-socket\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.719434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.692778 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-data\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.719434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.693898 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istiod-ca-cert\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.719434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.697734 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-workload-socket\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.719434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.699650 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-workload-certs\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.719434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.699951 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-credential-socket\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.745561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.727850 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-envoy\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.778264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.778150 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-podinfo\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.789594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.786228 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-token\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.852563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.847424 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8wwq\" (UniqueName: \"kubernetes.io/projected/e837c3cb-1bf5-4774-a9e1-a378f67048c1-kube-api-access-j8wwq\") pod \"e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:40.866286 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:40.865053 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:41.538176 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:41.538137 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x"] Nov 03 20:11:41.542307 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:11:41.542257 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode837c3cb_1bf5_4774_a9e1_a378f67048c1.slice/crio-fc0296637e69fd21f2421b9f85442e29020707722d378ea2de6136a361a2390a WatchSource:0}: Error finding container fc0296637e69fd21f2421b9f85442e29020707722d378ea2de6136a361a2390a: Status 404 returned error can't find the container with id fc0296637e69fd21f2421b9f85442e29020707722d378ea2de6136a361a2390a Nov 03 20:11:41.546479 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:41.546442 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:11:41.546576 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:41.546513 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:11:41.546576 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:41.546559 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:11:41.886034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:41.886000 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" event={"ID":"e837c3cb-1bf5-4774-a9e1-a378f67048c1","Type":"ContainerStarted","Data":"3466384fa52a7d9f63105b8efc6ff2eeb561039b9a79b7adb4866f27c0ecbd39"} Nov 03 20:11:41.886225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:41.886054 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" event={"ID":"e837c3cb-1bf5-4774-a9e1-a378f67048c1","Type":"ContainerStarted","Data":"fc0296637e69fd21f2421b9f85442e29020707722d378ea2de6136a361a2390a"} Nov 03 20:11:41.915388 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:41.915335 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" podStartSLOduration=1.915321651 podStartE2EDuration="1.915321651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:41.907497116 +0000 UTC m=+1931.869875554" watchObservedRunningTime="2026-11-03 20:11:41.915321651 +0000 UTC m=+1931.877700090" Nov 03 20:11:42.868270 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:42.868193 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:42.874731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:42.874707 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:42.895666 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:42.895638 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:42.896680 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:42.896655 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:11:43.989378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:43.989256 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf"] Nov 03 20:11:44.002560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:44.001442 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9dc48-5dqmf"] Nov 03 20:11:44.212782 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:44.210651 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebb4dcd0-d282-43ba-919b-ccd559228fa9" path="/var/lib/kubelet/pods/ebb4dcd0-d282-43ba-919b-ccd559228fa9/volumes" Nov 03 20:11:50.855222 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:50.855180 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz"] Nov 03 20:11:50.855912 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:50.855440 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" podUID="4ebf23a0-415f-452a-a530-aee813434ba7" containerName="istio-proxy" containerID="cri-o://8a09a7f74b5eccd9a234185a51f94e5a6eccc38aa230d2ccf206624f8468062b" gracePeriod=30 Nov 03 20:11:50.866543 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:50.866497 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2"] Nov 03 20:11:50.866819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:50.866787 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" podUID="937bc0e2-ac02-4c96-83ca-21e0c9b82506" containerName="istio-proxy" containerID="cri-o://b18bc6e8eb6ccc67aea97df1905623ac5f167663ab445745387f2a7651d8664d" gracePeriod=30 Nov 03 20:11:56.432985 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.432958 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:56.516619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.516587 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-istio-data\") pod \"4ebf23a0-415f-452a-a530-aee813434ba7\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " Nov 03 20:11:56.516814 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.516631 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-workload-certs\") pod \"4ebf23a0-415f-452a-a530-aee813434ba7\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " Nov 03 20:11:56.516814 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.516692 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ebf23a0-415f-452a-a530-aee813434ba7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebf23a0-415f-452a-a530-aee813434ba7-istio-podinfo\") pod \"4ebf23a0-415f-452a-a530-aee813434ba7\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " Nov 03 20:11:56.516939 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.516813 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebf23a0-415f-452a-a530-aee813434ba7-kube-api-access-z7gwj\" (UniqueName: \"kubernetes.io/projected/4ebf23a0-415f-452a-a530-aee813434ba7-kube-api-access-z7gwj\") pod \"4ebf23a0-415f-452a-a530-aee813434ba7\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " Nov 03 20:11:56.516939 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.516898 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-credential-socket\") pod \"4ebf23a0-415f-452a-a530-aee813434ba7\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " Nov 03 20:11:56.516939 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.516905 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-workload-certs" pod "4ebf23a0-415f-452a-a530-aee813434ba7" (UID: "4ebf23a0-415f-452a-a530-aee813434ba7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:56.517417 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.517108 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebf23a0-415f-452a-a530-aee813434ba7-istio-token\" (UniqueName: \"kubernetes.io/projected/4ebf23a0-415f-452a-a530-aee813434ba7-istio-token\") pod \"4ebf23a0-415f-452a-a530-aee813434ba7\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " Nov 03 20:11:56.517417 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.517178 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ebf23a0-415f-452a-a530-aee813434ba7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebf23a0-415f-452a-a530-aee813434ba7-istiod-ca-cert\") pod \"4ebf23a0-415f-452a-a530-aee813434ba7\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " Nov 03 20:11:56.517417 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.517233 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-istio-envoy\") pod \"4ebf23a0-415f-452a-a530-aee813434ba7\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " Nov 03 20:11:56.517417 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.517258 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-workload-socket\") pod \"4ebf23a0-415f-452a-a530-aee813434ba7\" (UID: \"4ebf23a0-415f-452a-a530-aee813434ba7\") " Nov 03 20:11:56.517417 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.517337 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-credential-socket" pod "4ebf23a0-415f-452a-a530-aee813434ba7" (UID: "4ebf23a0-415f-452a-a530-aee813434ba7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:56.518577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.517743 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-workload-socket" pod "4ebf23a0-415f-452a-a530-aee813434ba7" (UID: "4ebf23a0-415f-452a-a530-aee813434ba7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:56.518577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.517796 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-istio-data" pod "4ebf23a0-415f-452a-a530-aee813434ba7" (UID: "4ebf23a0-415f-452a-a530-aee813434ba7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:56.518577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.517675 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebf23a0-415f-452a-a530-aee813434ba7-istiod-ca-cert" pod "4ebf23a0-415f-452a-a530-aee813434ba7" (UID: "4ebf23a0-415f-452a-a530-aee813434ba7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:11:56.518577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.518214 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.518577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.518231 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebf23a0-415f-452a-a530-aee813434ba7-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.518577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.518239 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.518577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.518249 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.518577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.518257 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.541028 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.541003 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebf23a0-415f-452a-a530-aee813434ba7-kube-api-access-z7gwj" pod "4ebf23a0-415f-452a-a530-aee813434ba7" (UID: "4ebf23a0-415f-452a-a530-aee813434ba7"). InnerVolumeSpecName "kube-api-access-z7gwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:56.541772 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.541701 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebf23a0-415f-452a-a530-aee813434ba7-istio-token" pod "4ebf23a0-415f-452a-a530-aee813434ba7" (UID: "4ebf23a0-415f-452a-a530-aee813434ba7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:56.541772 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.541731 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ebf23a0-415f-452a-a530-aee813434ba7-istio-podinfo" pod "4ebf23a0-415f-452a-a530-aee813434ba7" (UID: "4ebf23a0-415f-452a-a530-aee813434ba7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:11:56.542709 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.542686 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-istio-envoy" pod "4ebf23a0-415f-452a-a530-aee813434ba7" (UID: "4ebf23a0-415f-452a-a530-aee813434ba7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:56.552509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.552484 5111 generic.go:356] "Generic (PLEG): container finished" podID="4ebf23a0-415f-452a-a530-aee813434ba7" containerID="8a09a7f74b5eccd9a234185a51f94e5a6eccc38aa230d2ccf206624f8468062b" exitCode=0 Nov 03 20:11:56.552668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.552622 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" event={"ID":"4ebf23a0-415f-452a-a530-aee813434ba7","Type":"ContainerDied","Data":"8a09a7f74b5eccd9a234185a51f94e5a6eccc38aa230d2ccf206624f8468062b"} Nov 03 20:11:56.552668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.552661 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" event={"ID":"4ebf23a0-415f-452a-a530-aee813434ba7","Type":"ContainerDied","Data":"f926410577950d7480b67669d62450f1ebf7d30a688898010997a64169b398d1"} Nov 03 20:11:56.552800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.552680 5111 scope.go:122] "RemoveContainer" containerID="8a09a7f74b5eccd9a234185a51f94e5a6eccc38aa230d2ccf206624f8468062b" Nov 03 20:11:56.552800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.552798 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz" Nov 03 20:11:56.561817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.561713 5111 generic.go:356] "Generic (PLEG): container finished" podID="937bc0e2-ac02-4c96-83ca-21e0c9b82506" containerID="b18bc6e8eb6ccc67aea97df1905623ac5f167663ab445745387f2a7651d8664d" exitCode=0 Nov 03 20:11:56.562556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.562233 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" event={"ID":"937bc0e2-ac02-4c96-83ca-21e0c9b82506","Type":"ContainerDied","Data":"b18bc6e8eb6ccc67aea97df1905623ac5f167663ab445745387f2a7651d8664d"} Nov 03 20:11:56.595199 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.595170 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz"] Nov 03 20:11:56.599269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.599241 5111 scope.go:122] "RemoveContainer" containerID="8a09a7f74b5eccd9a234185a51f94e5a6eccc38aa230d2ccf206624f8468062b" Nov 03 20:11:56.599770 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:11:56.599647 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a09a7f74b5eccd9a234185a51f94e5a6eccc38aa230d2ccf206624f8468062b\": container with ID starting with 8a09a7f74b5eccd9a234185a51f94e5a6eccc38aa230d2ccf206624f8468062b not found: ID does not exist" containerID="8a09a7f74b5eccd9a234185a51f94e5a6eccc38aa230d2ccf206624f8468062b" Nov 03 20:11:56.599770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.599684 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a09a7f74b5eccd9a234185a51f94e5a6eccc38aa230d2ccf206624f8468062b"} err="failed to get container status \"8a09a7f74b5eccd9a234185a51f94e5a6eccc38aa230d2ccf206624f8468062b\": rpc error: code = NotFound desc = could not find container \"8a09a7f74b5eccd9a234185a51f94e5a6eccc38aa230d2ccf206624f8468062b\": container with ID starting with 8a09a7f74b5eccd9a234185a51f94e5a6eccc38aa230d2ccf206624f8468062b not found: ID does not exist" Nov 03 20:11:56.604023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.603998 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-r2tqz"] Nov 03 20:11:56.620368 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.620342 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebf23a0-415f-452a-a530-aee813434ba7-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.620496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.620372 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7gwj\" (UniqueName: \"kubernetes.io/projected/4ebf23a0-415f-452a-a530-aee813434ba7-kube-api-access-z7gwj\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.620496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.620389 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebf23a0-415f-452a-a530-aee813434ba7-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.620660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.620497 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebf23a0-415f-452a-a530-aee813434ba7-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.776781 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.776693 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:56.823438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.823379 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-workload-socket\") pod \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " Nov 03 20:11:56.823438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.823438 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-data\") pod \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " Nov 03 20:11:56.823791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.823675 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937bc0e2-ac02-4c96-83ca-21e0c9b82506-kube-api-access-xh927\" (UniqueName: \"kubernetes.io/projected/937bc0e2-ac02-4c96-83ca-21e0c9b82506-kube-api-access-xh927\") pod \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " Nov 03 20:11:56.823862 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.823789 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-credential-socket\") pod \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " Nov 03 20:11:56.823862 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.823829 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-workload-certs\") pod \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " Nov 03 20:11:56.824005 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.823948 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istiod-ca-cert\") pod \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " Nov 03 20:11:56.824005 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.823978 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-token\" (UniqueName: \"kubernetes.io/projected/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-token\") pod \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " Nov 03 20:11:56.824148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.824026 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-envoy\") pod \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " Nov 03 20:11:56.824210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.824145 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-podinfo\") pod \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\" (UID: \"937bc0e2-ac02-4c96-83ca-21e0c9b82506\") " Nov 03 20:11:56.824392 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.824312 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-workload-socket" pod "937bc0e2-ac02-4c96-83ca-21e0c9b82506" (UID: "937bc0e2-ac02-4c96-83ca-21e0c9b82506"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:56.824735 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.824715 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-workload-certs" pod "937bc0e2-ac02-4c96-83ca-21e0c9b82506" (UID: "937bc0e2-ac02-4c96-83ca-21e0c9b82506"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:56.825042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.825023 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-data" pod "937bc0e2-ac02-4c96-83ca-21e0c9b82506" (UID: "937bc0e2-ac02-4c96-83ca-21e0c9b82506"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:56.825202 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.825180 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.825269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.825211 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.825485 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.825465 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-credential-socket" pod "937bc0e2-ac02-4c96-83ca-21e0c9b82506" (UID: "937bc0e2-ac02-4c96-83ca-21e0c9b82506"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:56.825916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.825890 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istiod-ca-cert" pod "937bc0e2-ac02-4c96-83ca-21e0c9b82506" (UID: "937bc0e2-ac02-4c96-83ca-21e0c9b82506"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:11:56.828085 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.828060 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937bc0e2-ac02-4c96-83ca-21e0c9b82506-kube-api-access-xh927" pod "937bc0e2-ac02-4c96-83ca-21e0c9b82506" (UID: "937bc0e2-ac02-4c96-83ca-21e0c9b82506"). InnerVolumeSpecName "kube-api-access-xh927". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:56.828187 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.828170 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-envoy" pod "937bc0e2-ac02-4c96-83ca-21e0c9b82506" (UID: "937bc0e2-ac02-4c96-83ca-21e0c9b82506"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:56.830816 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.830773 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-podinfo" pod "937bc0e2-ac02-4c96-83ca-21e0c9b82506" (UID: "937bc0e2-ac02-4c96-83ca-21e0c9b82506"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:11:56.832486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.832460 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-token" pod "937bc0e2-ac02-4c96-83ca-21e0c9b82506" (UID: "937bc0e2-ac02-4c96-83ca-21e0c9b82506"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:56.916282 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.916248 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms"] Nov 03 20:11:56.917128 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.917109 5111 cpu_manager.go:414] "containerMap: removing container" podUID="937bc0e2-ac02-4c96-83ca-21e0c9b82506" containerName="istio-proxy" Nov 03 20:11:56.917128 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.917126 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937bc0e2-ac02-4c96-83ca-21e0c9b82506" containerName="istio-proxy" Nov 03 20:11:56.917302 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.917135 5111 cpu_manager.go:414] "containerMap: removing container" podUID="4ebf23a0-415f-452a-a530-aee813434ba7" containerName="istio-proxy" Nov 03 20:11:56.917302 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.917141 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebf23a0-415f-452a-a530-aee813434ba7" containerName="istio-proxy" Nov 03 20:11:56.917415 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.917304 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="937bc0e2-ac02-4c96-83ca-21e0c9b82506" containerName="istio-proxy" Nov 03 20:11:56.917415 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.917311 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebf23a0-415f-452a-a530-aee813434ba7" containerName="istio-proxy" Nov 03 20:11:56.922047 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.922026 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" Nov 03 20:11:56.929006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.928977 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.929173 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.929158 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.929273 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.929260 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.929360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.929349 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.929449 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.929429 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.929617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.929451 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937bc0e2-ac02-4c96-83ca-21e0c9b82506-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.929617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.929468 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh927\" (UniqueName: \"kubernetes.io/projected/937bc0e2-ac02-4c96-83ca-21e0c9b82506-kube-api-access-xh927\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:56.931440 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.931420 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-43b780-dockercfg-bvw6l\"" Nov 03 20:11:56.932053 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.931675 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-43b780\"" Nov 03 20:11:56.933124 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.933100 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms"] Nov 03 20:11:56.941424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.941403 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92"] Nov 03 20:11:56.948500 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.948478 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" Nov 03 20:11:56.961066 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:56.960996 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92"] Nov 03 20:11:57.089605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.089407 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7b3d839-ce01-41bd-91fa-e1771134c8d8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-43b780-b75f87544-ghs92\" (UID: \"e7b3d839-ce01-41bd-91fa-e1771134c8d8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" Nov 03 20:11:57.089605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.089481 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98bb8ea8-1345-4d95-80cf-43ac69ad4344-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms\" (UID: \"98bb8ea8-1345-4d95-80cf-43ac69ad4344\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" Nov 03 20:11:57.089860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.089658 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hzm4\" (UniqueName: \"kubernetes.io/projected/98bb8ea8-1345-4d95-80cf-43ac69ad4344-kube-api-access-7hzm4\") pod \"payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms\" (UID: \"98bb8ea8-1345-4d95-80cf-43ac69ad4344\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" Nov 03 20:11:57.089860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.089834 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8glc\" (UniqueName: \"kubernetes.io/projected/e7b3d839-ce01-41bd-91fa-e1771134c8d8-kube-api-access-c8glc\") pod \"payload-processing-e2e-inf-a-43b780-b75f87544-ghs92\" (UID: \"e7b3d839-ce01-41bd-91fa-e1771134c8d8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" Nov 03 20:11:57.123318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.122170 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g"] Nov 03 20:11:57.262556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.252406 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8glc\" (UniqueName: \"kubernetes.io/projected/e7b3d839-ce01-41bd-91fa-e1771134c8d8-kube-api-access-c8glc\") pod \"payload-processing-e2e-inf-a-43b780-b75f87544-ghs92\" (UID: \"e7b3d839-ce01-41bd-91fa-e1771134c8d8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" Nov 03 20:11:57.262556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.252794 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7b3d839-ce01-41bd-91fa-e1771134c8d8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-43b780-b75f87544-ghs92\" (UID: \"e7b3d839-ce01-41bd-91fa-e1771134c8d8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" Nov 03 20:11:57.262556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.252885 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98bb8ea8-1345-4d95-80cf-43ac69ad4344-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms\" (UID: \"98bb8ea8-1345-4d95-80cf-43ac69ad4344\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" Nov 03 20:11:57.262556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.253099 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hzm4\" (UniqueName: \"kubernetes.io/projected/98bb8ea8-1345-4d95-80cf-43ac69ad4344-kube-api-access-7hzm4\") pod \"payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms\" (UID: \"98bb8ea8-1345-4d95-80cf-43ac69ad4344\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" Nov 03 20:11:57.262556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.262391 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98bb8ea8-1345-4d95-80cf-43ac69ad4344-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms\" (UID: \"98bb8ea8-1345-4d95-80cf-43ac69ad4344\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" Nov 03 20:11:57.292559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.289454 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7b3d839-ce01-41bd-91fa-e1771134c8d8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-43b780-b75f87544-ghs92\" (UID: \"e7b3d839-ce01-41bd-91fa-e1771134c8d8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" Nov 03 20:11:57.441628 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.426249 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hzm4\" (UniqueName: \"kubernetes.io/projected/98bb8ea8-1345-4d95-80cf-43ac69ad4344-kube-api-access-7hzm4\") pod \"payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms\" (UID: \"98bb8ea8-1345-4d95-80cf-43ac69ad4344\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" Nov 03 20:11:57.441628 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.440693 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8glc\" (UniqueName: \"kubernetes.io/projected/e7b3d839-ce01-41bd-91fa-e1771134c8d8-kube-api-access-c8glc\") pod \"payload-processing-e2e-inf-a-43b780-b75f87544-ghs92\" (UID: \"e7b3d839-ce01-41bd-91fa-e1771134c8d8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" Nov 03 20:11:57.452132 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.452097 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2418922c-mct8g"] Nov 03 20:11:57.544280 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.544246 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" Nov 03 20:11:57.581408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.581378 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" Nov 03 20:11:57.584698 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.584669 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" Nov 03 20:11:57.584698 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.584688 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2" event={"ID":"937bc0e2-ac02-4c96-83ca-21e0c9b82506","Type":"ContainerDied","Data":"66d140ea1a41d0ac74196f071acca91594474f538b8e947302c2393636d91a63"} Nov 03 20:11:57.585566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.584753 5111 scope.go:122] "RemoveContainer" containerID="b18bc6e8eb6ccc67aea97df1905623ac5f167663ab445745387f2a7651d8664d" Nov 03 20:11:57.649013 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.648985 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2"] Nov 03 20:11:57.654662 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.654637 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-h2gx2"] Nov 03 20:11:57.866061 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:11:57.866027 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bb8ea8_1345_4d95_80cf_43ac69ad4344.slice/crio-64ef770bcc8268d3a4a87b3469e5bddd4125c1f2f18a5bff76f91d64dac0ddd9 WatchSource:0}: Error finding container 64ef770bcc8268d3a4a87b3469e5bddd4125c1f2f18a5bff76f91d64dac0ddd9: Status 404 returned error can't find the container with id 64ef770bcc8268d3a4a87b3469e5bddd4125c1f2f18a5bff76f91d64dac0ddd9 Nov 03 20:11:57.866825 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.866745 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms"] Nov 03 20:11:57.910881 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:11:57.910848 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b3d839_ce01_41bd_91fa_e1771134c8d8.slice/crio-6181f3bbd50fdc40d08c8a4f41e67088c69b1f802a1d00b4db04e8763d54aaf2 WatchSource:0}: Error finding container 6181f3bbd50fdc40d08c8a4f41e67088c69b1f802a1d00b4db04e8763d54aaf2: Status 404 returned error can't find the container with id 6181f3bbd50fdc40d08c8a4f41e67088c69b1f802a1d00b4db04e8763d54aaf2 Nov 03 20:11:57.914864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.913228 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92"] Nov 03 20:11:57.929364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.929334 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw"] Nov 03 20:11:57.939061 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.939028 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" Nov 03 20:11:57.948589 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.947939 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw"] Nov 03 20:11:57.994766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.994744 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-facc79\"" Nov 03 20:11:57.995638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:57.995621 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-facc79-dockercfg-grffh\"" Nov 03 20:11:58.167776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.069439 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae340c1-f2b0-4d4f-ac20-8c2886e894b9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw\" (UID: \"fae340c1-f2b0-4d4f-ac20-8c2886e894b9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" Nov 03 20:11:58.167776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.069634 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snp4w\" (UniqueName: \"kubernetes.io/projected/fae340c1-f2b0-4d4f-ac20-8c2886e894b9-kube-api-access-snp4w\") pod \"payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw\" (UID: \"fae340c1-f2b0-4d4f-ac20-8c2886e894b9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" Nov 03 20:11:58.332686 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.330905 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae340c1-f2b0-4d4f-ac20-8c2886e894b9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw\" (UID: \"fae340c1-f2b0-4d4f-ac20-8c2886e894b9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" Nov 03 20:11:58.356579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.267692 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae340c1-f2b0-4d4f-ac20-8c2886e894b9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw\" (UID: \"fae340c1-f2b0-4d4f-ac20-8c2886e894b9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" Nov 03 20:11:58.356579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.333100 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snp4w\" (UniqueName: \"kubernetes.io/projected/fae340c1-f2b0-4d4f-ac20-8c2886e894b9-kube-api-access-snp4w\") pod \"payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw\" (UID: \"fae340c1-f2b0-4d4f-ac20-8c2886e894b9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" Nov 03 20:11:58.390681 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.390063 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snp4w\" (UniqueName: \"kubernetes.io/projected/fae340c1-f2b0-4d4f-ac20-8c2886e894b9-kube-api-access-snp4w\") pod \"payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw\" (UID: \"fae340c1-f2b0-4d4f-ac20-8c2886e894b9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" Nov 03 20:11:58.462897 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.462109 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ebf23a0-415f-452a-a530-aee813434ba7" path="/var/lib/kubelet/pods/4ebf23a0-415f-452a-a530-aee813434ba7/volumes" Nov 03 20:11:58.464319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.464294 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="937bc0e2-ac02-4c96-83ca-21e0c9b82506" path="/var/lib/kubelet/pods/937bc0e2-ac02-4c96-83ca-21e0c9b82506/volumes" Nov 03 20:11:58.465900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.465880 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed464a2-06ae-4d29-9880-9bbf2773ac1d" path="/var/lib/kubelet/pods/aed464a2-06ae-4d29-9880-9bbf2773ac1d/volumes" Nov 03 20:11:58.562889 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.562856 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" Nov 03 20:11:58.634597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.634552 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" event={"ID":"98bb8ea8-1345-4d95-80cf-43ac69ad4344","Type":"ContainerStarted","Data":"572f5404053d8969713a9bb8fdb264930ea235734fcdbd0cbf19a5238a91aa3a"} Nov 03 20:11:58.634801 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.634651 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" Nov 03 20:11:58.634801 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.634669 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" event={"ID":"98bb8ea8-1345-4d95-80cf-43ac69ad4344","Type":"ContainerStarted","Data":"64ef770bcc8268d3a4a87b3469e5bddd4125c1f2f18a5bff76f91d64dac0ddd9"} Nov 03 20:11:58.636826 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.636801 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" event={"ID":"e7b3d839-ce01-41bd-91fa-e1771134c8d8","Type":"ContainerStarted","Data":"6284dc7fe7153a22090afeb26b33740bac59da7e861827bc2b03258abd30b797"} Nov 03 20:11:58.636906 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.636842 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" event={"ID":"e7b3d839-ce01-41bd-91fa-e1771134c8d8","Type":"ContainerStarted","Data":"6181f3bbd50fdc40d08c8a4f41e67088c69b1f802a1d00b4db04e8763d54aaf2"} Nov 03 20:11:58.637617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.637596 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" Nov 03 20:11:58.657023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.656978 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" podStartSLOduration=2.656963428 podStartE2EDuration="2.656963428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:58.649914193 +0000 UTC m=+1948.612292632" watchObservedRunningTime="2026-11-03 20:11:58.656963428 +0000 UTC m=+1948.619341865" Nov 03 20:11:58.670711 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.670242 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" podStartSLOduration=2.670225779 podStartE2EDuration="2.670225779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:58.667216854 +0000 UTC m=+1948.629595294" watchObservedRunningTime="2026-11-03 20:11:58.670225779 +0000 UTC m=+1948.632604221" Nov 03 20:11:58.888493 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:58.888452 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw"] Nov 03 20:11:58.891831 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:11:58.891799 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae340c1_f2b0_4d4f_ac20_8c2886e894b9.slice/crio-1c46a350e8b9e15977bab4740a67c3f9d62e2dfebd3b4e48cb9b065aa9899b5a WatchSource:0}: Error finding container 1c46a350e8b9e15977bab4740a67c3f9d62e2dfebd3b4e48cb9b065aa9899b5a: Status 404 returned error can't find the container with id 1c46a350e8b9e15977bab4740a67c3f9d62e2dfebd3b4e48cb9b065aa9899b5a Nov 03 20:11:59.656111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:59.656066 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" event={"ID":"fae340c1-f2b0-4d4f-ac20-8c2886e894b9","Type":"ContainerStarted","Data":"c73792e820641395a277024b14e4395fd4c5f5e12026ccd4c7d6ba2bda766b88"} Nov 03 20:11:59.656697 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:59.656128 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" event={"ID":"fae340c1-f2b0-4d4f-ac20-8c2886e894b9","Type":"ContainerStarted","Data":"1c46a350e8b9e15977bab4740a67c3f9d62e2dfebd3b4e48cb9b065aa9899b5a"} Nov 03 20:11:59.656697 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:59.656562 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" Nov 03 20:11:59.678824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:11:59.678768 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" podStartSLOduration=2.678754038 podStartE2EDuration="2.678754038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:59.672802501 +0000 UTC m=+1949.635180960" watchObservedRunningTime="2026-11-03 20:11:59.678754038 +0000 UTC m=+1949.641132535" Nov 03 20:12:03.689122 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:03.689082 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66"] Nov 03 20:12:05.064310 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:12:05.064277 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:12:08.691434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:08.690626 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" podUID="0e61e6c5-045f-43e5-a944-a7f3aa50522b" containerName="payload-pre-processing" containerID="cri-o://f690f2b6309a6d9b0804ed9b2507d7e379bb9694312225222bcc3faf44e7091f" gracePeriod=25 Nov 03 20:12:08.829287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:08.829243 5111 generic.go:356] "Generic (PLEG): container finished" podID="0e61e6c5-045f-43e5-a944-a7f3aa50522b" containerID="f690f2b6309a6d9b0804ed9b2507d7e379bb9694312225222bcc3faf44e7091f" exitCode=0 Nov 03 20:12:08.829439 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:08.829429 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" event={"ID":"0e61e6c5-045f-43e5-a944-a7f3aa50522b","Type":"ContainerDied","Data":"f690f2b6309a6d9b0804ed9b2507d7e379bb9694312225222bcc3faf44e7091f"} Nov 03 20:12:09.040525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:09.040495 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" Nov 03 20:12:09.155821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:09.155790 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e61e6c5-045f-43e5-a944-a7f3aa50522b-kube-api-access-ldrct\" (UniqueName: \"kubernetes.io/projected/0e61e6c5-045f-43e5-a944-a7f3aa50522b-kube-api-access-ldrct\") pod \"0e61e6c5-045f-43e5-a944-a7f3aa50522b\" (UID: \"0e61e6c5-045f-43e5-a944-a7f3aa50522b\") " Nov 03 20:12:09.156087 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:09.156074 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e61e6c5-045f-43e5-a944-a7f3aa50522b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e61e6c5-045f-43e5-a944-a7f3aa50522b-plugins-config-volume\") pod \"0e61e6c5-045f-43e5-a944-a7f3aa50522b\" (UID: \"0e61e6c5-045f-43e5-a944-a7f3aa50522b\") " Nov 03 20:12:09.156461 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:09.156434 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e61e6c5-045f-43e5-a944-a7f3aa50522b-plugins-config-volume" pod "0e61e6c5-045f-43e5-a944-a7f3aa50522b" (UID: "0e61e6c5-045f-43e5-a944-a7f3aa50522b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:09.156924 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:09.156906 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e61e6c5-045f-43e5-a944-a7f3aa50522b-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:09.158003 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:09.157976 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e61e6c5-045f-43e5-a944-a7f3aa50522b-kube-api-access-ldrct" pod "0e61e6c5-045f-43e5-a944-a7f3aa50522b" (UID: "0e61e6c5-045f-43e5-a944-a7f3aa50522b"). InnerVolumeSpecName "kube-api-access-ldrct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:09.259494 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:09.259414 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldrct\" (UniqueName: \"kubernetes.io/projected/0e61e6c5-045f-43e5-a944-a7f3aa50522b-kube-api-access-ldrct\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:09.660298 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:09.660271 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" Nov 03 20:12:09.840841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:09.840813 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" Nov 03 20:12:09.841293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:09.840850 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66" event={"ID":"0e61e6c5-045f-43e5-a944-a7f3aa50522b","Type":"ContainerDied","Data":"08113c86d09f966b1476f30583aa1495816a83023d4f5bec133b8f55f33be479"} Nov 03 20:12:09.841293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:09.840917 5111 scope.go:122] "RemoveContainer" containerID="f690f2b6309a6d9b0804ed9b2507d7e379bb9694312225222bcc3faf44e7091f" Nov 03 20:12:09.868456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:09.868430 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66"] Nov 03 20:12:09.873229 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:09.873207 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-xxt66"] Nov 03 20:12:10.198977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:10.198939 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e61e6c5-045f-43e5-a944-a7f3aa50522b" path="/var/lib/kubelet/pods/0e61e6c5-045f-43e5-a944-a7f3aa50522b/volumes" Nov 03 20:12:10.755558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:10.754768 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" Nov 03 20:12:11.188901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.188864 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl"] Nov 03 20:12:11.190087 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.190057 5111 cpu_manager.go:414] "containerMap: removing container" podUID="0e61e6c5-045f-43e5-a944-a7f3aa50522b" containerName="payload-pre-processing" Nov 03 20:12:11.190087 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.190085 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e61e6c5-045f-43e5-a944-a7f3aa50522b" containerName="payload-pre-processing" Nov 03 20:12:11.190285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.190274 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e61e6c5-045f-43e5-a944-a7f3aa50522b" containerName="payload-pre-processing" Nov 03 20:12:11.195873 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.195830 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:12:11.205702 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.205515 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:12:11.206388 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.206362 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7bzpm\"" Nov 03 20:12:11.206716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.206698 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:12:11.206836 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.206763 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-433102-serving-cert\"" Nov 03 20:12:11.206939 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.206811 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:12:11.209437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.209417 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl"] Nov 03 20:12:11.303731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.303688 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b231a30b-b938-468b-ae10-1eccb1456524-metrics-certs\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-z9bpl\" (UID: \"b231a30b-b938-468b-ae10-1eccb1456524\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:12:11.303971 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.303952 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h6kb\" (UniqueName: \"kubernetes.io/projected/b231a30b-b938-468b-ae10-1eccb1456524-kube-api-access-9h6kb\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-z9bpl\" (UID: \"b231a30b-b938-468b-ae10-1eccb1456524\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:12:11.304226 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.304211 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b231a30b-b938-468b-ae10-1eccb1456524-openshift-service-ca\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-z9bpl\" (UID: \"b231a30b-b938-468b-ae10-1eccb1456524\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:12:11.304437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.304411 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b231a30b-b938-468b-ae10-1eccb1456524-maas-api-tls\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-z9bpl\" (UID: \"b231a30b-b938-468b-ae10-1eccb1456524\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:12:11.417708 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.414003 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h6kb\" (UniqueName: \"kubernetes.io/projected/b231a30b-b938-468b-ae10-1eccb1456524-kube-api-access-9h6kb\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-z9bpl\" (UID: \"b231a30b-b938-468b-ae10-1eccb1456524\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:12:11.417708 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.414141 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b231a30b-b938-468b-ae10-1eccb1456524-openshift-service-ca\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-z9bpl\" (UID: \"b231a30b-b938-468b-ae10-1eccb1456524\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:12:11.417708 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.414271 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b231a30b-b938-468b-ae10-1eccb1456524-maas-api-tls\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-z9bpl\" (UID: \"b231a30b-b938-468b-ae10-1eccb1456524\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:12:11.417708 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.414445 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b231a30b-b938-468b-ae10-1eccb1456524-metrics-certs\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-z9bpl\" (UID: \"b231a30b-b938-468b-ae10-1eccb1456524\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:12:11.423553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.419452 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b231a30b-b938-468b-ae10-1eccb1456524-openshift-service-ca\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-z9bpl\" (UID: \"b231a30b-b938-468b-ae10-1eccb1456524\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:12:11.447607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.441385 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h6kb\" (UniqueName: \"kubernetes.io/projected/b231a30b-b938-468b-ae10-1eccb1456524-kube-api-access-9h6kb\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-z9bpl\" (UID: \"b231a30b-b938-468b-ae10-1eccb1456524\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:12:11.466572 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.459359 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b231a30b-b938-468b-ae10-1eccb1456524-maas-api-tls\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-z9bpl\" (UID: \"b231a30b-b938-468b-ae10-1eccb1456524\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:12:11.477291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.477257 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b231a30b-b938-468b-ae10-1eccb1456524-metrics-certs\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-z9bpl\" (UID: \"b231a30b-b938-468b-ae10-1eccb1456524\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:12:11.524800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.524766 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:12:11.785074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:11.784096 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" Nov 03 20:12:12.452487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:12.449688 5111 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:12:12.487966 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:12.487932 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl"] Nov 03 20:12:12.894370 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:12.894334 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" event={"ID":"b231a30b-b938-468b-ae10-1eccb1456524","Type":"ContainerStarted","Data":"4bb0901c08d2986e7c5b1da6659bf42f8c860135ce717cce2d582a886d0a5f32"} Nov 03 20:12:14.924340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:14.924303 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" event={"ID":"b231a30b-b938-468b-ae10-1eccb1456524","Type":"ContainerStarted","Data":"4296d49a9505f63ccead646f1414537a9839e374d4488b499fc24e158075ad02"} Nov 03 20:12:14.924810 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:14.924402 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:12:14.947045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:14.946998 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" podStartSLOduration=1.628721885 podStartE2EDuration="3.9469839s" totalImagesPullingTime="2.318262015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:14.941972323 +0000 UTC m=+1964.904350774" watchObservedRunningTime="2026-11-03 20:12:14.9469839 +0000 UTC m=+1964.909362339" Nov 03 20:12:20.984755 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:20.984721 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:12:31.789692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:31.789652 5111 scope.go:122] "RemoveContainer" containerID="0cb132693d178a51d1a9e22b832844247c2690e85bd0eaa00c6c58d1eeea26fc" Nov 03 20:12:31.837639 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:31.837610 5111 scope.go:122] "RemoveContainer" containerID="1b6ad5e3d968fadf93077cca3da8d0fc44b3f01f254739dde3c5e48d2d0fdb1e" Nov 03 20:12:31.899850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:31.899818 5111 scope.go:122] "RemoveContainer" containerID="f1f83b03e29a1eb932c2cef92195f904374c5dd37e7f562a110e5969098babef" Nov 03 20:12:31.919729 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:31.919709 5111 scope.go:122] "RemoveContainer" containerID="aca86c5d2619eafec6b4963a8ffb3a48f323e8d7ea224c7f1a278510f237be9f" Nov 03 20:12:31.934515 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:31.934497 5111 scope.go:122] "RemoveContainer" containerID="5a714145ee047a6c52e00b2dc71bb5271a72c082324bd7419e4a19d36dca5887" Nov 03 20:12:32.045909 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:32.045706 5111 scope.go:122] "RemoveContainer" containerID="0d9a5ab2b49abfb88fda7efe2a7a9acd88af8bc1980067a3de96382173ad0cb2" Nov 03 20:12:32.469596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:32.465323 5111 scope.go:122] "RemoveContainer" containerID="8ba3921ac17d1ca1247fe90a397f0e35a75e6b80075699cde6f0205e7af15c77" Nov 03 20:12:32.740012 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:32.739985 5111 scope.go:122] "RemoveContainer" containerID="d231b7b03c619efc3960516ffae050b6ab5363bf65a2edd491159be1af8c4903" Nov 03 20:12:32.884802 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:32.884142 5111 scope.go:122] "RemoveContainer" containerID="e11653af039a9c83b227ef10618355af74e6af16f487fff068db6dbcfaab22ab" Nov 03 20:12:32.974854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:32.973146 5111 scope.go:122] "RemoveContainer" containerID="7fb2f22d8c56444ecabf3bba9cfae798e784fb943c4d25ef4e1d01a26fbbbaf5" Nov 03 20:12:33.057854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:33.053702 5111 scope.go:122] "RemoveContainer" containerID="61666ee3fb78465588429543abae2dbd2d20b13b5abc502b489f4537f110fe12" Nov 03 20:12:33.084733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:33.084709 5111 scope.go:122] "RemoveContainer" containerID="a14e8538fc6cc000b7242e144102cdb4f9c0acf8332593449dfa05ff473e135e" Nov 03 20:12:33.097364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:33.097342 5111 scope.go:122] "RemoveContainer" containerID="f342039b462ce05ed70473e4b7667ed8ce011d812dc69fc8e2a6f4c91a6aa092" Nov 03 20:12:33.108689 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:33.108668 5111 scope.go:122] "RemoveContainer" containerID="6e4b021992152632fbfcdf755b2ba00f3ea62347438ae1e76ae570f88c76a69f" Nov 03 20:12:33.120648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:33.120624 5111 scope.go:122] "RemoveContainer" containerID="a09a49d56097c64ebeac3a64efe70b0c840bd3694d59ed034d55dddf6ee0da3a" Nov 03 20:12:33.131186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:33.131169 5111 scope.go:122] "RemoveContainer" containerID="e175e427733c88e790141b5c4bae5e02ef236d33772f2ac2ba895fb3c9b0ed09" Nov 03 20:12:33.142215 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:33.142199 5111 scope.go:122] "RemoveContainer" containerID="15633b3c95a55849a44597dc0066188ca70fedcb43841c093ee882720349b5b3" Nov 03 20:12:33.183568 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:33.183544 5111 scope.go:122] "RemoveContainer" containerID="cfd29e38c66babfecee912cf037add8e1f5f1596a6a6437b1b78763d70a061bd" Nov 03 20:12:33.196266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:33.196237 5111 scope.go:122] "RemoveContainer" containerID="d22832cf73518f31f9ceedfd55f315590a93d6301fcdc08dab2a5147ed5aab52" Nov 03 20:12:33.212913 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:33.212890 5111 scope.go:122] "RemoveContainer" containerID="af50fef6d72621b9b1cde5712f2677a5ebf06fca89a753025c4954e9032ecde9" Nov 03 20:12:33.224970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:33.224947 5111 scope.go:122] "RemoveContainer" containerID="51eaeb3fafc316ca0e9ed18103022e7e8016b21572ce144a39d3a27320ff5d15" Nov 03 20:12:35.097875 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:12:35.097699 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:12:37.120848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:37.120811 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd"] Nov 03 20:12:37.127819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:37.127793 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc"] Nov 03 20:12:38.415958 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:38.350560 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm"] Nov 03 20:12:38.415958 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:38.350913 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" podUID="2a40c5e8-4cd0-4e49-8a87-9b578902940b" containerName="istio-proxy" containerID="cri-o://6094075973e28c7300334d3d09882f60c70f021592cef901d963daa8bd92d782" gracePeriod=30 Nov 03 20:12:42.121322 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.121260 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" podUID="80dfc8d1-743e-4c51-941f-c9dd6f279f76" containerName="payload-processing" containerID="cri-o://b1479e3e56437f506354575edcc917a7b620305883779383451fea54dee74dfa" gracePeriod=25 Nov 03 20:12:42.128625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.128591 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" podUID="05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f" containerName="payload-pre-processing" containerID="cri-o://00aac47e0ff690d70bc4c53d724c7e38916bc0f43db94250cdce94211abe64ac" gracePeriod=25 Nov 03 20:12:42.684004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.683971 5111 generic.go:356] "Generic (PLEG): container finished" podID="e837c3cb-1bf5-4774-a9e1-a378f67048c1" containerID="3466384fa52a7d9f63105b8efc6ff2eeb561039b9a79b7adb4866f27c0ecbd39" exitCode=0 Nov 03 20:12:42.684158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.684117 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" event={"ID":"e837c3cb-1bf5-4774-a9e1-a378f67048c1","Type":"ContainerDied","Data":"3466384fa52a7d9f63105b8efc6ff2eeb561039b9a79b7adb4866f27c0ecbd39"} Nov 03 20:12:42.684719 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.684701 5111 scope.go:122] "RemoveContainer" containerID="3466384fa52a7d9f63105b8efc6ff2eeb561039b9a79b7adb4866f27c0ecbd39" Nov 03 20:12:42.687939 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.687915 5111 generic.go:356] "Generic (PLEG): container finished" podID="80dfc8d1-743e-4c51-941f-c9dd6f279f76" containerID="b1479e3e56437f506354575edcc917a7b620305883779383451fea54dee74dfa" exitCode=0 Nov 03 20:12:42.688111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.688012 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" event={"ID":"80dfc8d1-743e-4c51-941f-c9dd6f279f76","Type":"ContainerDied","Data":"b1479e3e56437f506354575edcc917a7b620305883779383451fea54dee74dfa"} Nov 03 20:12:42.690670 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.690580 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:12:42.690670 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.690656 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:12:42.690808 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.690696 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:12:42.693026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.691878 5111 generic.go:356] "Generic (PLEG): container finished" podID="05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f" containerID="00aac47e0ff690d70bc4c53d724c7e38916bc0f43db94250cdce94211abe64ac" exitCode=0 Nov 03 20:12:42.693026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.691948 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" event={"ID":"05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f","Type":"ContainerDied","Data":"00aac47e0ff690d70bc4c53d724c7e38916bc0f43db94250cdce94211abe64ac"} Nov 03 20:12:42.793500 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.793472 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" Nov 03 20:12:42.797403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.796796 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" Nov 03 20:12:42.898080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.874650 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v"] Nov 03 20:12:42.898080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.882066 5111 cpu_manager.go:414] "containerMap: removing container" podUID="05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f" containerName="payload-pre-processing" Nov 03 20:12:42.898080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.882092 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f" containerName="payload-pre-processing" Nov 03 20:12:42.898080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.882166 5111 cpu_manager.go:414] "containerMap: removing container" podUID="80dfc8d1-743e-4c51-941f-c9dd6f279f76" containerName="payload-processing" Nov 03 20:12:42.898080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.882176 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80dfc8d1-743e-4c51-941f-c9dd6f279f76" containerName="payload-processing" Nov 03 20:12:42.898080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.882743 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="80dfc8d1-743e-4c51-941f-c9dd6f279f76" containerName="payload-processing" Nov 03 20:12:42.898080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.882769 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f" containerName="payload-pre-processing" Nov 03 20:12:42.898080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.886118 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" Nov 03 20:12:42.898080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.889169 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v"] Nov 03 20:12:42.898080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.891921 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f817b0-dockercfg-664v5\"" Nov 03 20:12:42.898080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.891989 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f817b0\"" Nov 03 20:12:42.898080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.896089 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:12:42.987552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.942275 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80dfc8d1-743e-4c51-941f-c9dd6f279f76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dfc8d1-743e-4c51-941f-c9dd6f279f76-plugins-config-volume\") pod \"80dfc8d1-743e-4c51-941f-c9dd6f279f76\" (UID: \"80dfc8d1-743e-4c51-941f-c9dd6f279f76\") " Nov 03 20:12:42.987552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.942665 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f-plugins-config-volume\") pod \"05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f\" (UID: \"05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f\") " Nov 03 20:12:42.987552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.942713 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80dfc8d1-743e-4c51-941f-c9dd6f279f76-kube-api-access-fcfnq\" (UniqueName: \"kubernetes.io/projected/80dfc8d1-743e-4c51-941f-c9dd6f279f76-kube-api-access-fcfnq\") pod \"80dfc8d1-743e-4c51-941f-c9dd6f279f76\" (UID: \"80dfc8d1-743e-4c51-941f-c9dd6f279f76\") " Nov 03 20:12:42.987552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.942907 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f-kube-api-access-zlqb9\" (UniqueName: \"kubernetes.io/projected/05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f-kube-api-access-zlqb9\") pod \"05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f\" (UID: \"05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f\") " Nov 03 20:12:42.987552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.954359 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f-plugins-config-volume" pod "05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f" (UID: "05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:42.987552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.954834 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80dfc8d1-743e-4c51-941f-c9dd6f279f76-plugins-config-volume" pod "80dfc8d1-743e-4c51-941f-c9dd6f279f76" (UID: "80dfc8d1-743e-4c51-941f-c9dd6f279f76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:42.987552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.958258 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dfc8d1-743e-4c51-941f-c9dd6f279f76-kube-api-access-fcfnq" pod "80dfc8d1-743e-4c51-941f-c9dd6f279f76" (UID: "80dfc8d1-743e-4c51-941f-c9dd6f279f76"). InnerVolumeSpecName "kube-api-access-fcfnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:42.992769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:42.992736 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f-kube-api-access-zlqb9" pod "05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f" (UID: "05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f"). InnerVolumeSpecName "kube-api-access-zlqb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:43.048512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.046946 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4dpg\" (UniqueName: \"kubernetes.io/projected/96f02f41-3bee-4567-a64f-4ccc94f07b32-kube-api-access-n4dpg\") pod \"payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v\" (UID: \"96f02f41-3bee-4567-a64f-4ccc94f07b32\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" Nov 03 20:12:43.048512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.047250 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f02f41-3bee-4567-a64f-4ccc94f07b32-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v\" (UID: \"96f02f41-3bee-4567-a64f-4ccc94f07b32\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" Nov 03 20:12:43.079621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.072655 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:43.079621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.072692 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcfnq\" (UniqueName: \"kubernetes.io/projected/80dfc8d1-743e-4c51-941f-c9dd6f279f76-kube-api-access-fcfnq\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:43.079621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.072709 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlqb9\" (UniqueName: \"kubernetes.io/projected/05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f-kube-api-access-zlqb9\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:43.079621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.072776 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80dfc8d1-743e-4c51-941f-c9dd6f279f76-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:43.176560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.174351 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4dpg\" (UniqueName: \"kubernetes.io/projected/96f02f41-3bee-4567-a64f-4ccc94f07b32-kube-api-access-n4dpg\") pod \"payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v\" (UID: \"96f02f41-3bee-4567-a64f-4ccc94f07b32\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" Nov 03 20:12:43.176560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.174673 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f02f41-3bee-4567-a64f-4ccc94f07b32-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v\" (UID: \"96f02f41-3bee-4567-a64f-4ccc94f07b32\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" Nov 03 20:12:43.177180 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.176711 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f02f41-3bee-4567-a64f-4ccc94f07b32-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v\" (UID: \"96f02f41-3bee-4567-a64f-4ccc94f07b32\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" Nov 03 20:12:43.212314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.207935 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4dpg\" (UniqueName: \"kubernetes.io/projected/96f02f41-3bee-4567-a64f-4ccc94f07b32-kube-api-access-n4dpg\") pod \"payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v\" (UID: \"96f02f41-3bee-4567-a64f-4ccc94f07b32\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" Nov 03 20:12:43.525612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.521936 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" Nov 03 20:12:43.737488 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.737454 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" event={"ID":"80dfc8d1-743e-4c51-941f-c9dd6f279f76","Type":"ContainerDied","Data":"8585aa345933e28d0e46c903bbe6589a5fb413028805682165f7efa704b074c4"} Nov 03 20:12:43.737805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.737526 5111 scope.go:122] "RemoveContainer" containerID="b1479e3e56437f506354575edcc917a7b620305883779383451fea54dee74dfa" Nov 03 20:12:43.737805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.737692 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd" Nov 03 20:12:43.807639 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.804875 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" event={"ID":"05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f","Type":"ContainerDied","Data":"6e88cd19e378e41392749d720b53395d92f0a0acd85124f95c6158a902c0ff6d"} Nov 03 20:12:43.807639 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.804994 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc" Nov 03 20:12:43.827160 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.827017 5111 scope.go:122] "RemoveContainer" containerID="00aac47e0ff690d70bc4c53d724c7e38916bc0f43db94250cdce94211abe64ac" Nov 03 20:12:43.835982 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.835961 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd"] Nov 03 20:12:43.837132 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.837109 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" event={"ID":"e837c3cb-1bf5-4774-a9e1-a378f67048c1","Type":"ContainerStarted","Data":"4001bd6d523ac51f0c372783bc1b7c4ff42926e3d5277235f9487ba4cdb2d82b"} Nov 03 20:12:43.842753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.842731 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-433102-b5fc7466d-j6pdd"] Nov 03 20:12:43.902329 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.902162 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:12:43.906006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.905971 5111 patch_prober.go:29] interesting pod/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:12:43.906152 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.906035 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" podUID="e837c3cb-1bf5-4774-a9e1-a378f67048c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:12:43.908206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.908050 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc"] Nov 03 20:12:43.917633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:43.917583 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-433102-7ccd48d8f9-5xjsc"] Nov 03 20:12:44.225278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.225239 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f" path="/var/lib/kubelet/pods/05e9b1fb-1d9a-4a20-b79b-5236d1ffde3f/volumes" Nov 03 20:12:44.226619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.226597 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80dfc8d1-743e-4c51-941f-c9dd6f279f76" path="/var/lib/kubelet/pods/80dfc8d1-743e-4c51-941f-c9dd6f279f76/volumes" Nov 03 20:12:44.315989 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.315958 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v"] Nov 03 20:12:44.523852 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.523822 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:12:44.620848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.620810 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-credential-socket\") pod \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " Nov 03 20:12:44.621381 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.621226 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-credential-socket" pod "2a40c5e8-4cd0-4e49-8a87-9b578902940b" (UID: "2a40c5e8-4cd0-4e49-8a87-9b578902940b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:44.621381 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.621370 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istiod-ca-cert\") pod \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " Nov 03 20:12:44.621636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.621405 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-envoy\") pod \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " Nov 03 20:12:44.621636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.621585 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-podinfo\") pod \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " Nov 03 20:12:44.621744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.621692 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-data\") pod \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " Nov 03 20:12:44.621782 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.621757 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-workload-certs\") pod \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " Nov 03 20:12:44.622077 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.621846 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-token\" (UniqueName: \"kubernetes.io/projected/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-token\") pod \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " Nov 03 20:12:44.622077 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.621934 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-workload-socket\") pod \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " Nov 03 20:12:44.622077 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.621988 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a40c5e8-4cd0-4e49-8a87-9b578902940b-kube-api-access-7v9cn\" (UniqueName: \"kubernetes.io/projected/2a40c5e8-4cd0-4e49-8a87-9b578902940b-kube-api-access-7v9cn\") pod \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\" (UID: \"2a40c5e8-4cd0-4e49-8a87-9b578902940b\") " Nov 03 20:12:44.622325 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.622258 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-workload-certs" pod "2a40c5e8-4cd0-4e49-8a87-9b578902940b" (UID: "2a40c5e8-4cd0-4e49-8a87-9b578902940b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:44.622325 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.622299 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-workload-socket" pod "2a40c5e8-4cd0-4e49-8a87-9b578902940b" (UID: "2a40c5e8-4cd0-4e49-8a87-9b578902940b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:44.622520 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.622499 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-data" pod "2a40c5e8-4cd0-4e49-8a87-9b578902940b" (UID: "2a40c5e8-4cd0-4e49-8a87-9b578902940b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:44.623068 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.623038 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istiod-ca-cert" pod "2a40c5e8-4cd0-4e49-8a87-9b578902940b" (UID: "2a40c5e8-4cd0-4e49-8a87-9b578902940b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:44.623665 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.623641 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:44.623759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.623693 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:44.623759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.623703 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:44.623759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.623713 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:44.623759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.623721 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:44.624750 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.624714 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-podinfo" pod "2a40c5e8-4cd0-4e49-8a87-9b578902940b" (UID: "2a40c5e8-4cd0-4e49-8a87-9b578902940b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:12:44.625702 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.625667 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a40c5e8-4cd0-4e49-8a87-9b578902940b-kube-api-access-7v9cn" pod "2a40c5e8-4cd0-4e49-8a87-9b578902940b" (UID: "2a40c5e8-4cd0-4e49-8a87-9b578902940b"). InnerVolumeSpecName "kube-api-access-7v9cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:44.625804 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.625721 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-envoy" pod "2a40c5e8-4cd0-4e49-8a87-9b578902940b" (UID: "2a40c5e8-4cd0-4e49-8a87-9b578902940b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:44.625881 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.625845 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-token" pod "2a40c5e8-4cd0-4e49-8a87-9b578902940b" (UID: "2a40c5e8-4cd0-4e49-8a87-9b578902940b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:44.725765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.725731 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:44.725765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.725763 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v9cn\" (UniqueName: \"kubernetes.io/projected/2a40c5e8-4cd0-4e49-8a87-9b578902940b-kube-api-access-7v9cn\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:44.725765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.725774 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:44.725997 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.725783 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a40c5e8-4cd0-4e49-8a87-9b578902940b-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:44.871712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.871671 5111 patch_prober.go:29] interesting pod/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:12:44.871887 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.871737 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" podUID="e837c3cb-1bf5-4774-a9e1-a378f67048c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:12:44.875880 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.875847 5111 generic.go:356] "Generic (PLEG): container finished" podID="2a40c5e8-4cd0-4e49-8a87-9b578902940b" containerID="6094075973e28c7300334d3d09882f60c70f021592cef901d963daa8bd92d782" exitCode=0 Nov 03 20:12:44.875995 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.875897 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" Nov 03 20:12:44.875995 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.875930 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" event={"ID":"2a40c5e8-4cd0-4e49-8a87-9b578902940b","Type":"ContainerDied","Data":"6094075973e28c7300334d3d09882f60c70f021592cef901d963daa8bd92d782"} Nov 03 20:12:44.875995 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.875992 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm" event={"ID":"2a40c5e8-4cd0-4e49-8a87-9b578902940b","Type":"ContainerDied","Data":"5d00d59947e6e31dea803b2e14be1e0f38399aa1f993beb1e54b45122fe387a5"} Nov 03 20:12:44.876132 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.876006 5111 scope.go:122] "RemoveContainer" containerID="6094075973e28c7300334d3d09882f60c70f021592cef901d963daa8bd92d782" Nov 03 20:12:44.885327 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.885297 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" event={"ID":"96f02f41-3bee-4567-a64f-4ccc94f07b32","Type":"ContainerStarted","Data":"32a1ffc8b6bbc1a1b3aba809c63c0b6275b15f0c607912b5acd7245e5bac36c1"} Nov 03 20:12:44.885444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.885354 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" event={"ID":"96f02f41-3bee-4567-a64f-4ccc94f07b32","Type":"ContainerStarted","Data":"ce570a9a44bd670ac5ceba506058f86f9fed493812f04c3c0af3f975033c26f6"} Nov 03 20:12:44.885621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.885605 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" Nov 03 20:12:44.894248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.894226 5111 scope.go:122] "RemoveContainer" containerID="6094075973e28c7300334d3d09882f60c70f021592cef901d963daa8bd92d782" Nov 03 20:12:44.894511 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:12:44.894486 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6094075973e28c7300334d3d09882f60c70f021592cef901d963daa8bd92d782\": container with ID starting with 6094075973e28c7300334d3d09882f60c70f021592cef901d963daa8bd92d782 not found: ID does not exist" containerID="6094075973e28c7300334d3d09882f60c70f021592cef901d963daa8bd92d782" Nov 03 20:12:44.894640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.894515 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6094075973e28c7300334d3d09882f60c70f021592cef901d963daa8bd92d782"} err="failed to get container status \"6094075973e28c7300334d3d09882f60c70f021592cef901d963daa8bd92d782\": rpc error: code = NotFound desc = could not find container \"6094075973e28c7300334d3d09882f60c70f021592cef901d963daa8bd92d782\": container with ID starting with 6094075973e28c7300334d3d09882f60c70f021592cef901d963daa8bd92d782 not found: ID does not exist" Nov 03 20:12:44.910047 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:44.909997 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" podStartSLOduration=7.909980743 podStartE2EDuration="7.909980743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:44.90977206 +0000 UTC m=+1994.872150499" watchObservedRunningTime="2026-11-03 20:12:44.909980743 +0000 UTC m=+1994.872359187" Nov 03 20:12:45.033555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:45.024340 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm"] Nov 03 20:12:45.036169 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:45.034033 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-5j6zm"] Nov 03 20:12:45.870591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:45.869998 5111 patch_prober.go:29] interesting pod/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:12:45.870591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:45.870063 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" podUID="e837c3cb-1bf5-4774-a9e1-a378f67048c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:12:46.188976 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:46.188898 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a40c5e8-4cd0-4e49-8a87-9b578902940b" path="/var/lib/kubelet/pods/2a40c5e8-4cd0-4e49-8a87-9b578902940b/volumes" Nov 03 20:12:46.868697 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:46.868657 5111 patch_prober.go:29] interesting pod/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:12:46.869004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:46.868969 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" podUID="e837c3cb-1bf5-4774-a9e1-a378f67048c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:12:47.869527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:47.869267 5111 patch_prober.go:29] interesting pod/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:12:47.869527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:47.869329 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" podUID="e837c3cb-1bf5-4774-a9e1-a378f67048c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:12:48.872993 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:48.872427 5111 patch_prober.go:29] interesting pod/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:12:48.872993 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:48.872500 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" podUID="e837c3cb-1bf5-4774-a9e1-a378f67048c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:12:49.873953 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:49.873920 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:12:49.874646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:49.874341 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:12:49.875148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:49.875119 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:12:52.149477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:52.149446 5111 generic.go:356] "Generic (PLEG): container finished" podID="90eb063f-fb7a-4c71-b959-8504f519eb4e" containerID="c19adc674bea25870bbbba24bd3861386162d5ade016292a35c341536bc274c0" exitCode=0 Nov 03 20:12:52.150001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:52.149515 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" event={"ID":"90eb063f-fb7a-4c71-b959-8504f519eb4e","Type":"ContainerDied","Data":"c19adc674bea25870bbbba24bd3861386162d5ade016292a35c341536bc274c0"} Nov 03 20:12:52.150109 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:52.150094 5111 scope.go:122] "RemoveContainer" containerID="c19adc674bea25870bbbba24bd3861386162d5ade016292a35c341536bc274c0" Nov 03 20:12:52.152070 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:52.152044 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:12:52.152132 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:52.152112 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:12:52.152167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:52.152141 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:12:53.216806 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:53.202347 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" event={"ID":"90eb063f-fb7a-4c71-b959-8504f519eb4e","Type":"ContainerStarted","Data":"ba18c4525e1932c22b67b444d8354a9444faf55a2bc4d3d15ac85f0d72d448d4"} Nov 03 20:12:53.588588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:53.588498 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:12:53.588766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:53.588730 5111 patch_prober.go:29] interesting pod/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 20:12:53.588836 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:53.588790 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" podUID="90eb063f-fb7a-4c71-b959-8504f519eb4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 20:12:53.820377 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:53.820326 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66"] Nov 03 20:12:53.822740 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:53.822701 5111 cpu_manager.go:414] "containerMap: removing container" podUID="2a40c5e8-4cd0-4e49-8a87-9b578902940b" containerName="istio-proxy" Nov 03 20:12:53.822942 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:53.822928 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a40c5e8-4cd0-4e49-8a87-9b578902940b" containerName="istio-proxy" Nov 03 20:12:53.823402 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:53.823384 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a40c5e8-4cd0-4e49-8a87-9b578902940b" containerName="istio-proxy" Nov 03 20:12:53.828242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:53.828219 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66"] Nov 03 20:12:53.828386 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:53.828361 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:12:53.832093 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:53.832072 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f817b0-serving-cert\"" Nov 03 20:12:54.004908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:54.004870 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66\" (UID: \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:12:54.005095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:54.005071 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-metrics-certs\") pod \"maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66\" (UID: \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:12:54.005209 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:54.005194 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlp5r\" (UniqueName: \"kubernetes.io/projected/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-kube-api-access-wlp5r\") pod \"maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66\" (UID: \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:12:54.005266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:54.005226 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-maas-api-tls\") pod \"maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66\" (UID: \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:12:54.112758 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:54.107416 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66\" (UID: \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:12:54.112758 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:54.107602 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-metrics-certs\") pod \"maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66\" (UID: \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:12:54.112758 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:54.107710 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlp5r\" (UniqueName: \"kubernetes.io/projected/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-kube-api-access-wlp5r\") pod \"maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66\" (UID: \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:12:54.112758 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:54.107749 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-maas-api-tls\") pod \"maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66\" (UID: \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:12:54.112758 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:54.109800 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66\" (UID: \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:12:54.131431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:54.131396 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-maas-api-tls\") pod \"maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66\" (UID: \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:12:54.141353 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:54.136623 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-metrics-certs\") pod \"maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66\" (UID: \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:12:54.151188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:54.151159 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlp5r\" (UniqueName: \"kubernetes.io/projected/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-kube-api-access-wlp5r\") pod \"maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66\" (UID: \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:12:54.203263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:54.203231 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:12:54.619717 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:54.619680 5111 patch_prober.go:29] interesting pod/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 20:12:54.640757 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:54.640651 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" podUID="90eb063f-fb7a-4c71-b959-8504f519eb4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 20:12:54.772956 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:54.766911 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66"] Nov 03 20:12:54.797375 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:12:54.797334 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c13b0c_0fbf_47bd_a1e2_602608404ea3.slice/crio-d09ec808269f79a8f934e31fe353294ff049f72b3b59d6022c13eba5b77c4284 WatchSource:0}: Error finding container d09ec808269f79a8f934e31fe353294ff049f72b3b59d6022c13eba5b77c4284: Status 404 returned error can't find the container with id d09ec808269f79a8f934e31fe353294ff049f72b3b59d6022c13eba5b77c4284 Nov 03 20:12:55.338555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:55.335398 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" event={"ID":"a9c13b0c-0fbf-47bd-a1e2-602608404ea3","Type":"ContainerStarted","Data":"d09ec808269f79a8f934e31fe353294ff049f72b3b59d6022c13eba5b77c4284"} Nov 03 20:12:55.623554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:55.621968 5111 patch_prober.go:29] interesting pod/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 20:12:55.623554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:55.622030 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" podUID="90eb063f-fb7a-4c71-b959-8504f519eb4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 20:12:56.076569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:56.059754 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" Nov 03 20:12:56.355127 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:56.348481 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" event={"ID":"a9c13b0c-0fbf-47bd-a1e2-602608404ea3","Type":"ContainerStarted","Data":"24230e91165f14630a9115f1c999afa5b3f5d77f48a6e993fdac9032cb60b5da"} Nov 03 20:12:56.355127 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:56.349726 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:12:56.379148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:56.379095 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" podStartSLOduration=2.499205594 podStartE2EDuration="3.379081072s" totalImagesPullingTime="879.875478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:56.375355426 +0000 UTC m=+2006.337733899" watchObservedRunningTime="2026-11-03 20:12:56.379081072 +0000 UTC m=+2006.341459509" Nov 03 20:12:56.619133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:56.613938 5111 patch_prober.go:29] interesting pod/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 20:12:56.619133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:56.614013 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" podUID="90eb063f-fb7a-4c71-b959-8504f519eb4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 20:12:57.103686 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:57.103583 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk"] Nov 03 20:12:57.108887 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:57.108857 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d70263d-5fksk"] Nov 03 20:12:57.588627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:57.588586 5111 patch_prober.go:29] interesting pod/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 20:12:57.588822 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:57.588649 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" podUID="90eb063f-fb7a-4c71-b959-8504f519eb4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 20:12:58.189472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:58.189438 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd0382f-8e90-4ce8-b2de-7d710be227f5" path="/var/lib/kubelet/pods/cfd0382f-8e90-4ce8-b2de-7d710be227f5/volumes" Nov 03 20:12:58.569815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:58.569707 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:12:58.587808 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:58.587755 5111 patch_prober.go:29] interesting pod/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 20:12:58.587982 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:58.587805 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" podUID="90eb063f-fb7a-4c71-b959-8504f519eb4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 20:12:59.641553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:59.607650 5111 patch_prober.go:29] interesting pod/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 20:12:59.641553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:12:59.607721 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" podUID="90eb063f-fb7a-4c71-b959-8504f519eb4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 20:13:00.593075 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:00.593043 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:13:00.594446 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:00.594410 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:13:03.483554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:03.475787 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:13:05.063114 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:13:05.063071 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:13:07.471814 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:07.471780 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl"] Nov 03 20:13:07.472272 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:07.472036 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" podUID="b231a30b-b938-468b-ae10-1eccb1456524" containerName="maas-api" containerID="cri-o://4296d49a9505f63ccead646f1414537a9839e374d4488b499fc24e158075ad02" gracePeriod=30 Nov 03 20:13:08.178034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.176306 5111 generic.go:356] "Generic (PLEG): container finished" podID="b231a30b-b938-468b-ae10-1eccb1456524" containerID="4296d49a9505f63ccead646f1414537a9839e374d4488b499fc24e158075ad02" exitCode=0 Nov 03 20:13:08.238883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.238837 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" event={"ID":"b231a30b-b938-468b-ae10-1eccb1456524","Type":"ContainerDied","Data":"4296d49a9505f63ccead646f1414537a9839e374d4488b499fc24e158075ad02"} Nov 03 20:13:08.299768 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.296240 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:13:08.448854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.448765 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww"] Nov 03 20:13:08.449272 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.449244 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" podUID="a3172c55-00d5-4500-b74e-e0d0cf88cb2e" containerName="istio-proxy" containerID="cri-o://12917f41f64b6e44fa4671cd1f087fdb23955d3c1324ad8ae3a580a7735bbb0a" gracePeriod=30 Nov 03 20:13:08.454314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.454283 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b231a30b-b938-468b-ae10-1eccb1456524-kube-api-access-9h6kb\" (UniqueName: \"kubernetes.io/projected/b231a30b-b938-468b-ae10-1eccb1456524-kube-api-access-9h6kb\") pod \"b231a30b-b938-468b-ae10-1eccb1456524\" (UID: \"b231a30b-b938-468b-ae10-1eccb1456524\") " Nov 03 20:13:08.454411 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.454385 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b231a30b-b938-468b-ae10-1eccb1456524-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b231a30b-b938-468b-ae10-1eccb1456524-metrics-certs\") pod \"b231a30b-b938-468b-ae10-1eccb1456524\" (UID: \"b231a30b-b938-468b-ae10-1eccb1456524\") " Nov 03 20:13:08.454473 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.454460 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b231a30b-b938-468b-ae10-1eccb1456524-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b231a30b-b938-468b-ae10-1eccb1456524-openshift-service-ca\") pod \"b231a30b-b938-468b-ae10-1eccb1456524\" (UID: \"b231a30b-b938-468b-ae10-1eccb1456524\") " Nov 03 20:13:08.454511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.454489 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b231a30b-b938-468b-ae10-1eccb1456524-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b231a30b-b938-468b-ae10-1eccb1456524-maas-api-tls\") pod \"b231a30b-b938-468b-ae10-1eccb1456524\" (UID: \"b231a30b-b938-468b-ae10-1eccb1456524\") " Nov 03 20:13:08.455235 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.455187 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b231a30b-b938-468b-ae10-1eccb1456524-openshift-service-ca" pod "b231a30b-b938-468b-ae10-1eccb1456524" (UID: "b231a30b-b938-468b-ae10-1eccb1456524"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:08.456145 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.456113 5111 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b231a30b-b938-468b-ae10-1eccb1456524-openshift-service-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:08.457827 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.457793 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b231a30b-b938-468b-ae10-1eccb1456524-kube-api-access-9h6kb" pod "b231a30b-b938-468b-ae10-1eccb1456524" (UID: "b231a30b-b938-468b-ae10-1eccb1456524"). InnerVolumeSpecName "kube-api-access-9h6kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:08.467578 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.466400 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b231a30b-b938-468b-ae10-1eccb1456524-maas-api-tls" pod "b231a30b-b938-468b-ae10-1eccb1456524" (UID: "b231a30b-b938-468b-ae10-1eccb1456524"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:13:08.467578 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.466562 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b231a30b-b938-468b-ae10-1eccb1456524-metrics-certs" pod "b231a30b-b938-468b-ae10-1eccb1456524" (UID: "b231a30b-b938-468b-ae10-1eccb1456524"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:13:08.581587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.566362 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h6kb\" (UniqueName: \"kubernetes.io/projected/b231a30b-b938-468b-ae10-1eccb1456524-kube-api-access-9h6kb\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:08.582207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.582170 5111 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b231a30b-b938-468b-ae10-1eccb1456524-metrics-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:08.582341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.582328 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b231a30b-b938-468b-ae10-1eccb1456524-maas-api-tls\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:08.768763 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.768678 5111 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:13:08.769054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:08.769030 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" podUID="a3172c55-00d5-4500-b74e-e0d0cf88cb2e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:13:09.190260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:09.190229 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" event={"ID":"b231a30b-b938-468b-ae10-1eccb1456524","Type":"ContainerDied","Data":"4bb0901c08d2986e7c5b1da6659bf42f8c860135ce717cce2d582a886d0a5f32"} Nov 03 20:13:09.190260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:09.190250 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl" Nov 03 20:13:09.190648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:09.190280 5111 scope.go:122] "RemoveContainer" containerID="4296d49a9505f63ccead646f1414537a9839e374d4488b499fc24e158075ad02" Nov 03 20:13:09.222315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:09.222290 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl"] Nov 03 20:13:09.226988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:09.226967 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-z9bpl"] Nov 03 20:13:10.191265 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:10.191232 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b231a30b-b938-468b-ae10-1eccb1456524" path="/var/lib/kubelet/pods/b231a30b-b938-468b-ae10-1eccb1456524/volumes" Nov 03 20:13:14.432595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.431708 5111 generic.go:356] "Generic (PLEG): container finished" podID="a3172c55-00d5-4500-b74e-e0d0cf88cb2e" containerID="12917f41f64b6e44fa4671cd1f087fdb23955d3c1324ad8ae3a580a7735bbb0a" exitCode=0 Nov 03 20:13:14.432595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.431775 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" event={"ID":"a3172c55-00d5-4500-b74e-e0d0cf88cb2e","Type":"ContainerDied","Data":"12917f41f64b6e44fa4671cd1f087fdb23955d3c1324ad8ae3a580a7735bbb0a"} Nov 03 20:13:14.432595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.431822 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" event={"ID":"a3172c55-00d5-4500-b74e-e0d0cf88cb2e","Type":"ContainerDied","Data":"9fa6376f3ba083f60f25e75401533db4602619bc61d4d79e66bb2b6005d0ff7f"} Nov 03 20:13:14.432595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.431845 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fa6376f3ba083f60f25e75401533db4602619bc61d4d79e66bb2b6005d0ff7f" Nov 03 20:13:14.445582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.442930 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:13:14.547549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.532145 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-workload-certs\") pod \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " Nov 03 20:13:14.547549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.532218 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-credential-socket\") pod \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " Nov 03 20:13:14.547549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.532587 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istiod-ca-cert\") pod \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " Nov 03 20:13:14.547549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.532620 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-podinfo\") pod \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " Nov 03 20:13:14.547549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.532683 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-kube-api-access-pqqtv\" (UniqueName: \"kubernetes.io/projected/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-kube-api-access-pqqtv\") pod \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " Nov 03 20:13:14.547549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.532725 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-envoy\") pod \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " Nov 03 20:13:14.547549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.532888 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-workload-socket\") pod \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " Nov 03 20:13:14.547549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.532931 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-token\" (UniqueName: \"kubernetes.io/projected/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-token\") pod \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " Nov 03 20:13:14.547549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.532973 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-data\") pod \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\" (UID: \"a3172c55-00d5-4500-b74e-e0d0cf88cb2e\") " Nov 03 20:13:14.547549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.537417 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-data" pod "a3172c55-00d5-4500-b74e-e0d0cf88cb2e" (UID: "a3172c55-00d5-4500-b74e-e0d0cf88cb2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:13:14.547549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.537696 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-workload-certs" pod "a3172c55-00d5-4500-b74e-e0d0cf88cb2e" (UID: "a3172c55-00d5-4500-b74e-e0d0cf88cb2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:13:14.547549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.537917 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-credential-socket" pod "a3172c55-00d5-4500-b74e-e0d0cf88cb2e" (UID: "a3172c55-00d5-4500-b74e-e0d0cf88cb2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:13:14.547549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.538200 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istiod-ca-cert" pod "a3172c55-00d5-4500-b74e-e0d0cf88cb2e" (UID: "a3172c55-00d5-4500-b74e-e0d0cf88cb2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:14.567551 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.552798 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-workload-socket" pod "a3172c55-00d5-4500-b74e-e0d0cf88cb2e" (UID: "a3172c55-00d5-4500-b74e-e0d0cf88cb2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:13:14.590979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.590883 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-envoy" pod "a3172c55-00d5-4500-b74e-e0d0cf88cb2e" (UID: "a3172c55-00d5-4500-b74e-e0d0cf88cb2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:13:14.651806 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.639792 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:14.651806 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.639834 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:14.651806 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.639851 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:14.651806 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.639866 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:14.651806 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.639883 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:14.651806 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.639898 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:14.693693 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.693437 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-podinfo" pod "a3172c55-00d5-4500-b74e-e0d0cf88cb2e" (UID: "a3172c55-00d5-4500-b74e-e0d0cf88cb2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:13:14.693693 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.693650 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-token" pod "a3172c55-00d5-4500-b74e-e0d0cf88cb2e" (UID: "a3172c55-00d5-4500-b74e-e0d0cf88cb2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:14.694148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.694123 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-kube-api-access-pqqtv" pod "a3172c55-00d5-4500-b74e-e0d0cf88cb2e" (UID: "a3172c55-00d5-4500-b74e-e0d0cf88cb2e"). InnerVolumeSpecName "kube-api-access-pqqtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:14.776085 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.742350 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:14.776085 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.775514 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqqtv\" (UniqueName: \"kubernetes.io/projected/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-kube-api-access-pqqtv\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:14.776085 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:14.775577 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3172c55-00d5-4500-b74e-e0d0cf88cb2e-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:15.556559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:15.513026 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww" Nov 03 20:13:15.743641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:15.743602 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww"] Nov 03 20:13:15.965627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:15.965593 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-ldcww"] Nov 03 20:13:16.022558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.004567 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq"] Nov 03 20:13:16.022558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.004816 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" podUID="90eb063f-fb7a-4c71-b959-8504f519eb4e" containerName="istio-proxy" containerID="cri-o://ba18c4525e1932c22b67b444d8354a9444faf55a2bc4d3d15ac85f0d72d448d4" gracePeriod=30 Nov 03 20:13:16.490583 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.481317 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3172c55-00d5-4500-b74e-e0d0cf88cb2e" path="/var/lib/kubelet/pods/a3172c55-00d5-4500-b74e-e0d0cf88cb2e/volumes" Nov 03 20:13:16.564409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.564375 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r"] Nov 03 20:13:16.566457 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.566419 5111 cpu_manager.go:414] "containerMap: removing container" podUID="a3172c55-00d5-4500-b74e-e0d0cf88cb2e" containerName="istio-proxy" Nov 03 20:13:16.628429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.628224 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3172c55-00d5-4500-b74e-e0d0cf88cb2e" containerName="istio-proxy" Nov 03 20:13:16.628429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.628295 5111 cpu_manager.go:414] "containerMap: removing container" podUID="b231a30b-b938-468b-ae10-1eccb1456524" containerName="maas-api" Nov 03 20:13:16.628429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.628307 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b231a30b-b938-468b-ae10-1eccb1456524" containerName="maas-api" Nov 03 20:13:16.628755 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.628655 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="b231a30b-b938-468b-ae10-1eccb1456524" containerName="maas-api" Nov 03 20:13:16.628755 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.628704 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3172c55-00d5-4500-b74e-e0d0cf88cb2e" containerName="istio-proxy" Nov 03 20:13:16.641936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.641895 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:16.644796 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.644773 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" Nov 03 20:13:16.649830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.649655 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:13:16.649830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.649786 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:13:16.678555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.656513 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r"] Nov 03 20:13:16.729417 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.729299 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r\" (UID: \"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" Nov 03 20:13:16.730068 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.730038 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r\" (UID: \"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" Nov 03 20:13:16.730450 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.730425 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clkws\" (UniqueName: \"kubernetes.io/projected/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-kube-api-access-clkws\") pod \"maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r\" (UID: \"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" Nov 03 20:13:16.914686 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.914629 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:16.976437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:16.974762 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:17.026996 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.026950 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:17.054525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.053550 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 20:13:17.054525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.053825 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 20:13:17.054525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.054056 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 20:13:17.054525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.054228 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded" Nov 03 20:13:17.054525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.054280 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:13:17.058670 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.058642 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r\" (UID: \"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" Nov 03 20:13:17.097232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.097109 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r\" (UID: \"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" Nov 03 20:13:17.145866 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.059729 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r\" (UID: \"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" Nov 03 20:13:17.152619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.143962 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:17.210069 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.177221 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r\" (UID: \"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" Nov 03 20:13:17.210375 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.210337 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:17.357561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.356226 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clkws\" (UniqueName: \"kubernetes.io/projected/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-kube-api-access-clkws\") pod \"maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r\" (UID: \"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" Nov 03 20:13:17.414557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.409665 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 03 20:13:17.706556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.706436 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clkws\" (UniqueName: \"kubernetes.io/projected/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-kube-api-access-clkws\") pod \"maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r\" (UID: \"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" Nov 03 20:13:17.945556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.924493 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:17.972608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.962783 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:13:17.994567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:17.991962 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" Nov 03 20:13:18.064546 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:18.062426 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:18.064546 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:18.062719 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:18.064546 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:18.063263 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:13:18.064546 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:18.063335 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:13:18.168556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:18.161811 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:13:18.168556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:18.162436 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:13:18.172615 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:18.170826 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:13:18.227625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:18.194650 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:13:18.362347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:18.361794 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:13:18.362347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:18.361856 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:13:18.699116 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:18.699077 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r"] Nov 03 20:13:18.703462 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:13:18.703424 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2de4d0_c6bd_4cb8_9996_5ba8ece1d617.slice/crio-c044844162c02fb508eb33f7a0391d631d617fd510929b7fd3de85a788714eba WatchSource:0}: Error finding container c044844162c02fb508eb33f7a0391d631d617fd510929b7fd3de85a788714eba: Status 404 returned error can't find the container with id c044844162c02fb508eb33f7a0391d631d617fd510929b7fd3de85a788714eba Nov 03 20:13:18.852486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:18.852450 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" event={"ID":"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617","Type":"ContainerStarted","Data":"c044844162c02fb508eb33f7a0391d631d617fd510929b7fd3de85a788714eba"} Nov 03 20:13:19.863295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:19.863255 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" event={"ID":"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617","Type":"ContainerStarted","Data":"886493d544a5573fea9f1739fc4cb83042e5b0bab7297edae130681a1f4e89d4"} Nov 03 20:13:19.884049 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:19.883996 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" podStartSLOduration=3.883982544 podStartE2EDuration="3.883982544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:19.882828198 +0000 UTC m=+2029.845206716" watchObservedRunningTime="2026-11-03 20:13:19.883982544 +0000 UTC m=+2029.846360981" Nov 03 20:13:20.874031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:20.874001 5111 generic.go:356] "Generic (PLEG): container finished" podID="2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617" containerID="886493d544a5573fea9f1739fc4cb83042e5b0bab7297edae130681a1f4e89d4" exitCode=0 Nov 03 20:13:20.874476 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:20.874085 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" event={"ID":"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617","Type":"ContainerDied","Data":"886493d544a5573fea9f1739fc4cb83042e5b0bab7297edae130681a1f4e89d4"} Nov 03 20:13:21.398302 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.398272 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:13:21.563576 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.563234 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-credential-socket\") pod \"90eb063f-fb7a-4c71-b959-8504f519eb4e\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " Nov 03 20:13:21.563576 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.563298 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-token\" (UniqueName: \"kubernetes.io/projected/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-token\") pod \"90eb063f-fb7a-4c71-b959-8504f519eb4e\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " Nov 03 20:13:21.563576 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.563354 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90eb063f-fb7a-4c71-b959-8504f519eb4e-kube-api-access-tzswn\" (UniqueName: \"kubernetes.io/projected/90eb063f-fb7a-4c71-b959-8504f519eb4e-kube-api-access-tzswn\") pod \"90eb063f-fb7a-4c71-b959-8504f519eb4e\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " Nov 03 20:13:21.563871 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.563637 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-workload-certs\") pod \"90eb063f-fb7a-4c71-b959-8504f519eb4e\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " Nov 03 20:13:21.563871 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.563750 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-data\") pod \"90eb063f-fb7a-4c71-b959-8504f519eb4e\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " Nov 03 20:13:21.563871 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.563788 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-workload-socket\") pod \"90eb063f-fb7a-4c71-b959-8504f519eb4e\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " Nov 03 20:13:21.563871 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.563861 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-podinfo\") pod \"90eb063f-fb7a-4c71-b959-8504f519eb4e\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " Nov 03 20:13:21.564241 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.563888 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90eb063f-fb7a-4c71-b959-8504f519eb4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90eb063f-fb7a-4c71-b959-8504f519eb4e-istiod-ca-cert\") pod \"90eb063f-fb7a-4c71-b959-8504f519eb4e\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " Nov 03 20:13:21.564241 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.563959 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-envoy\") pod \"90eb063f-fb7a-4c71-b959-8504f519eb4e\" (UID: \"90eb063f-fb7a-4c71-b959-8504f519eb4e\") " Nov 03 20:13:21.564891 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.564864 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-workload-certs" pod "90eb063f-fb7a-4c71-b959-8504f519eb4e" (UID: "90eb063f-fb7a-4c71-b959-8504f519eb4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:13:21.565236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.565214 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-credential-socket" pod "90eb063f-fb7a-4c71-b959-8504f519eb4e" (UID: "90eb063f-fb7a-4c71-b959-8504f519eb4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:13:21.571282 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.569927 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-token" pod "90eb063f-fb7a-4c71-b959-8504f519eb4e" (UID: "90eb063f-fb7a-4c71-b959-8504f519eb4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:21.579116 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.577601 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-data" pod "90eb063f-fb7a-4c71-b959-8504f519eb4e" (UID: "90eb063f-fb7a-4c71-b959-8504f519eb4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:13:21.590521 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.579412 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90eb063f-fb7a-4c71-b959-8504f519eb4e-istiod-ca-cert" pod "90eb063f-fb7a-4c71-b959-8504f519eb4e" (UID: "90eb063f-fb7a-4c71-b959-8504f519eb4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:21.590521 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.579645 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-workload-socket" pod "90eb063f-fb7a-4c71-b959-8504f519eb4e" (UID: "90eb063f-fb7a-4c71-b959-8504f519eb4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:13:21.590521 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.590155 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90eb063f-fb7a-4c71-b959-8504f519eb4e-kube-api-access-tzswn" pod "90eb063f-fb7a-4c71-b959-8504f519eb4e" (UID: "90eb063f-fb7a-4c71-b959-8504f519eb4e"). InnerVolumeSpecName "kube-api-access-tzswn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:21.601186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.600923 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzswn\" (UniqueName: \"kubernetes.io/projected/90eb063f-fb7a-4c71-b959-8504f519eb4e-kube-api-access-tzswn\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:21.601186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.600958 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:21.601186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.600975 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:21.601186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.600989 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:21.601186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.601001 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90eb063f-fb7a-4c71-b959-8504f519eb4e-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:21.601186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.601013 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:21.601186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.601026 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:21.601740 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.601711 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-envoy" pod "90eb063f-fb7a-4c71-b959-8504f519eb4e" (UID: "90eb063f-fb7a-4c71-b959-8504f519eb4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:13:21.601815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.601795 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-podinfo" pod "90eb063f-fb7a-4c71-b959-8504f519eb4e" (UID: "90eb063f-fb7a-4c71-b959-8504f519eb4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:13:21.706006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.705959 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:21.706006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.705998 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90eb063f-fb7a-4c71-b959-8504f519eb4e-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:21.894959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.894884 5111 generic.go:356] "Generic (PLEG): container finished" podID="90eb063f-fb7a-4c71-b959-8504f519eb4e" containerID="ba18c4525e1932c22b67b444d8354a9444faf55a2bc4d3d15ac85f0d72d448d4" exitCode=0 Nov 03 20:13:21.895409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.894959 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" Nov 03 20:13:21.895409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.895014 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" event={"ID":"90eb063f-fb7a-4c71-b959-8504f519eb4e","Type":"ContainerDied","Data":"ba18c4525e1932c22b67b444d8354a9444faf55a2bc4d3d15ac85f0d72d448d4"} Nov 03 20:13:21.895409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.895112 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq" event={"ID":"90eb063f-fb7a-4c71-b959-8504f519eb4e","Type":"ContainerDied","Data":"89e22a4cf6f6092b7be4830846bf78ac4e3dea4658e49055292859a5f791c1a1"} Nov 03 20:13:21.895409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.895147 5111 scope.go:122] "RemoveContainer" containerID="ba18c4525e1932c22b67b444d8354a9444faf55a2bc4d3d15ac85f0d72d448d4" Nov 03 20:13:21.910765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.910745 5111 scope.go:122] "RemoveContainer" containerID="c19adc674bea25870bbbba24bd3861386162d5ade016292a35c341536bc274c0" Nov 03 20:13:21.926845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.926818 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq"] Nov 03 20:13:21.931918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.931598 5111 scope.go:122] "RemoveContainer" containerID="ba18c4525e1932c22b67b444d8354a9444faf55a2bc4d3d15ac85f0d72d448d4" Nov 03 20:13:21.932278 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:13:21.932253 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba18c4525e1932c22b67b444d8354a9444faf55a2bc4d3d15ac85f0d72d448d4\": container with ID starting with ba18c4525e1932c22b67b444d8354a9444faf55a2bc4d3d15ac85f0d72d448d4 not found: ID does not exist" containerID="ba18c4525e1932c22b67b444d8354a9444faf55a2bc4d3d15ac85f0d72d448d4" Nov 03 20:13:21.932438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.932412 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba18c4525e1932c22b67b444d8354a9444faf55a2bc4d3d15ac85f0d72d448d4"} err="failed to get container status \"ba18c4525e1932c22b67b444d8354a9444faf55a2bc4d3d15ac85f0d72d448d4\": rpc error: code = NotFound desc = could not find container \"ba18c4525e1932c22b67b444d8354a9444faf55a2bc4d3d15ac85f0d72d448d4\": container with ID starting with ba18c4525e1932c22b67b444d8354a9444faf55a2bc4d3d15ac85f0d72d448d4 not found: ID does not exist" Nov 03 20:13:21.932563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.932548 5111 scope.go:122] "RemoveContainer" containerID="c19adc674bea25870bbbba24bd3861386162d5ade016292a35c341536bc274c0" Nov 03 20:13:21.933083 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:13:21.933015 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c19adc674bea25870bbbba24bd3861386162d5ade016292a35c341536bc274c0\": container with ID starting with c19adc674bea25870bbbba24bd3861386162d5ade016292a35c341536bc274c0 not found: ID does not exist" containerID="c19adc674bea25870bbbba24bd3861386162d5ade016292a35c341536bc274c0" Nov 03 20:13:21.933083 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.933047 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c19adc674bea25870bbbba24bd3861386162d5ade016292a35c341536bc274c0"} err="failed to get container status \"c19adc674bea25870bbbba24bd3861386162d5ade016292a35c341536bc274c0\": rpc error: code = NotFound desc = could not find container \"c19adc674bea25870bbbba24bd3861386162d5ade016292a35c341536bc274c0\": container with ID starting with c19adc674bea25870bbbba24bd3861386162d5ade016292a35c341536bc274c0 not found: ID does not exist" Nov 03 20:13:21.933592 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:21.933560 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-433102-openshift-default-555967f4f5-hjlkq"] Nov 03 20:13:22.314240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:22.313990 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90eb063f-fb7a-4c71-b959-8504f519eb4e" path="/var/lib/kubelet/pods/90eb063f-fb7a-4c71-b959-8504f519eb4e/volumes" Nov 03 20:13:22.349703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:22.349360 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" Nov 03 20:13:22.556612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:22.556097 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-maas-api-xks-ca\") pod \"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617\" (UID: \"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617\") " Nov 03 20:13:22.556612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:22.556342 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-maas-api-openshift-ca\") pod \"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617\" (UID: \"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617\") " Nov 03 20:13:22.556612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:22.556433 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-kube-api-access-clkws\" (UniqueName: \"kubernetes.io/projected/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-kube-api-access-clkws\") pod \"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617\" (UID: \"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617\") " Nov 03 20:13:22.570073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:22.558319 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-maas-api-openshift-ca" pod "2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617" (UID: "2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:22.570073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:22.562548 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-kube-api-access-clkws" pod "2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617" (UID: "2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617"). InnerVolumeSpecName "kube-api-access-clkws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:22.600002 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:22.599915 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-maas-api-xks-ca" pod "2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617" (UID: "2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:13:22.662557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:22.661438 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:22.662557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:22.661468 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clkws\" (UniqueName: \"kubernetes.io/projected/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-kube-api-access-clkws\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:22.662557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:22.661478 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:22.913561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:22.912929 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" event={"ID":"2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617","Type":"ContainerDied","Data":"c044844162c02fb508eb33f7a0391d631d617fd510929b7fd3de85a788714eba"} Nov 03 20:13:22.913561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:22.912999 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c044844162c02fb508eb33f7a0391d631d617fd510929b7fd3de85a788714eba" Nov 03 20:13:22.913561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:22.913081 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r" Nov 03 20:13:24.072108 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.072071 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz"] Nov 03 20:13:24.073527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.073499 5111 cpu_manager.go:414] "containerMap: removing container" podUID="90eb063f-fb7a-4c71-b959-8504f519eb4e" containerName="istio-proxy" Nov 03 20:13:24.073716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.073694 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90eb063f-fb7a-4c71-b959-8504f519eb4e" containerName="istio-proxy" Nov 03 20:13:24.073871 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.073854 5111 cpu_manager.go:414] "containerMap: removing container" podUID="2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617" containerName="revoke-keys" Nov 03 20:13:24.073973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.073960 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617" containerName="revoke-keys" Nov 03 20:13:24.074115 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.074094 5111 cpu_manager.go:414] "containerMap: removing container" podUID="90eb063f-fb7a-4c71-b959-8504f519eb4e" containerName="istio-proxy" Nov 03 20:13:24.074115 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.074112 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90eb063f-fb7a-4c71-b959-8504f519eb4e" containerName="istio-proxy" Nov 03 20:13:24.074414 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.074399 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="90eb063f-fb7a-4c71-b959-8504f519eb4e" containerName="istio-proxy" Nov 03 20:13:24.074450 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.074438 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="90eb063f-fb7a-4c71-b959-8504f519eb4e" containerName="istio-proxy" Nov 03 20:13:24.074483 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.074447 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617" containerName="revoke-keys" Nov 03 20:13:24.082989 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.082967 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" Nov 03 20:13:24.089234 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.089206 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz"] Nov 03 20:13:24.097313 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.097282 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq"] Nov 03 20:13:24.104702 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.104665 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" Nov 03 20:13:24.108183 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.107702 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff84f5c-617a-4dcf-8bda-ba203f11f648-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz\" (UID: \"eff84f5c-617a-4dcf-8bda-ba203f11f648\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" Nov 03 20:13:24.108183 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.107838 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4hmj\" (UniqueName: \"kubernetes.io/projected/eff84f5c-617a-4dcf-8bda-ba203f11f648-kube-api-access-d4hmj\") pod \"payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz\" (UID: \"eff84f5c-617a-4dcf-8bda-ba203f11f648\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" Nov 03 20:13:24.111964 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.111900 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq"] Nov 03 20:13:24.113490 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.113469 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-babea4-dockercfg-g4w6b\"" Nov 03 20:13:24.136761 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.136729 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-babea4\"" Nov 03 20:13:24.211175 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.211128 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff84f5c-617a-4dcf-8bda-ba203f11f648-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz\" (UID: \"eff84f5c-617a-4dcf-8bda-ba203f11f648\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" Nov 03 20:13:24.211344 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.211256 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4hmj\" (UniqueName: \"kubernetes.io/projected/eff84f5c-617a-4dcf-8bda-ba203f11f648-kube-api-access-d4hmj\") pod \"payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz\" (UID: \"eff84f5c-617a-4dcf-8bda-ba203f11f648\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" Nov 03 20:13:24.211478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.211457 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7xf8\" (UniqueName: \"kubernetes.io/projected/f6017f42-4ca1-4feb-a33a-c2eaf1585d8d-kube-api-access-h7xf8\") pod \"payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq\" (UID: \"f6017f42-4ca1-4feb-a33a-c2eaf1585d8d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" Nov 03 20:13:24.211594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.211576 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6017f42-4ca1-4feb-a33a-c2eaf1585d8d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq\" (UID: \"f6017f42-4ca1-4feb-a33a-c2eaf1585d8d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" Nov 03 20:13:24.212740 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.212714 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff84f5c-617a-4dcf-8bda-ba203f11f648-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz\" (UID: \"eff84f5c-617a-4dcf-8bda-ba203f11f648\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" Nov 03 20:13:24.238739 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.229743 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4hmj\" (UniqueName: \"kubernetes.io/projected/eff84f5c-617a-4dcf-8bda-ba203f11f648-kube-api-access-d4hmj\") pod \"payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz\" (UID: \"eff84f5c-617a-4dcf-8bda-ba203f11f648\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" Nov 03 20:13:24.380633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.375583 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7xf8\" (UniqueName: \"kubernetes.io/projected/f6017f42-4ca1-4feb-a33a-c2eaf1585d8d-kube-api-access-h7xf8\") pod \"payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq\" (UID: \"f6017f42-4ca1-4feb-a33a-c2eaf1585d8d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" Nov 03 20:13:24.380633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.375727 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6017f42-4ca1-4feb-a33a-c2eaf1585d8d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq\" (UID: \"f6017f42-4ca1-4feb-a33a-c2eaf1585d8d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" Nov 03 20:13:24.380633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.377120 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6017f42-4ca1-4feb-a33a-c2eaf1585d8d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq\" (UID: \"f6017f42-4ca1-4feb-a33a-c2eaf1585d8d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" Nov 03 20:13:24.418563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.418253 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" Nov 03 20:13:24.437562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.431146 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7xf8\" (UniqueName: \"kubernetes.io/projected/f6017f42-4ca1-4feb-a33a-c2eaf1585d8d-kube-api-access-h7xf8\") pod \"payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq\" (UID: \"f6017f42-4ca1-4feb-a33a-c2eaf1585d8d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" Nov 03 20:13:24.737945 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.737907 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" Nov 03 20:13:24.948131 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:24.946956 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz"] Nov 03 20:13:25.030425 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:25.030389 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" event={"ID":"eff84f5c-617a-4dcf-8bda-ba203f11f648","Type":"ContainerStarted","Data":"5456df645417f45c796a2f603344dada1810786e4f47a79607860dea89cda2c7"} Nov 03 20:13:25.163592 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:25.163381 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq"] Nov 03 20:13:25.168154 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:13:25.168118 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6017f42_4ca1_4feb_a33a_c2eaf1585d8d.slice/crio-20a36cec00b40d9df3cdb2249223b7ce3ffd6d904e3a87c685d876d2b3945471 WatchSource:0}: Error finding container 20a36cec00b40d9df3cdb2249223b7ce3ffd6d904e3a87c685d876d2b3945471: Status 404 returned error can't find the container with id 20a36cec00b40d9df3cdb2249223b7ce3ffd6d904e3a87c685d876d2b3945471 Nov 03 20:13:26.044614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:26.044571 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" event={"ID":"eff84f5c-617a-4dcf-8bda-ba203f11f648","Type":"ContainerStarted","Data":"7186bc456140056b5b92e4e24c67de4757116df8cebb98d04151e646ecfe08b5"} Nov 03 20:13:26.044828 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:26.044722 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" Nov 03 20:13:26.047226 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:26.047191 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" event={"ID":"f6017f42-4ca1-4feb-a33a-c2eaf1585d8d","Type":"ContainerStarted","Data":"a2c8a83ad7e0d0f5ae42c85e4b22c62f9424cb7b779f58bccbbc4335aa0793fc"} Nov 03 20:13:26.047344 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:26.047301 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" Nov 03 20:13:26.047344 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:26.047332 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" event={"ID":"f6017f42-4ca1-4feb-a33a-c2eaf1585d8d","Type":"ContainerStarted","Data":"20a36cec00b40d9df3cdb2249223b7ce3ffd6d904e3a87c685d876d2b3945471"} Nov 03 20:13:26.065785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:26.065735 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" podStartSLOduration=2.065720853 podStartE2EDuration="2.065720853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:26.062829774 +0000 UTC m=+2036.025208213" watchObservedRunningTime="2026-11-03 20:13:26.065720853 +0000 UTC m=+2036.028099293" Nov 03 20:13:26.083405 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:26.083353 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" podStartSLOduration=2.083340891 podStartE2EDuration="2.083340891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:26.080684699 +0000 UTC m=+2036.043063137" watchObservedRunningTime="2026-11-03 20:13:26.083340891 +0000 UTC m=+2036.045719359" Nov 03 20:13:33.578948 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:33.578912 5111 scope.go:122] "RemoveContainer" containerID="8ea93620795f6f4b3b40a5ca3f8a911215de510d7a9fcbc784af36e15421143f" Nov 03 20:13:33.611127 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:33.611095 5111 scope.go:122] "RemoveContainer" containerID="66e026ea3277d37426559c87855a5f6efdaf8984df7a8fc0391047c571c3edc9" Nov 03 20:13:33.625450 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:33.625416 5111 scope.go:122] "RemoveContainer" containerID="33b34d57090caae9db412886304e8ee64463bf92cb70c9941012d7713bef3007" Nov 03 20:13:33.642230 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:33.642203 5111 scope.go:122] "RemoveContainer" containerID="bd17bfa3cf7b0297a876848ab264192a85a8bffbed32547a316895a2ba46a14a" Nov 03 20:13:34.240293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.240256 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5"] Nov 03 20:13:34.247732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.247698 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.254423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.254396 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-49fb2b-openshift-default-dockercfg-w4fn6\"" Nov 03 20:13:34.264944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.263744 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5"] Nov 03 20:13:34.312616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.312578 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-credential-socket\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.312827 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.312668 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-envoy\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.313136 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.312957 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2phrl\" (UniqueName: \"kubernetes.io/projected/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-kube-api-access-2phrl\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.313136 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.313050 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-podinfo\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.313287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.313214 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-workload-certs\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.313479 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.313458 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-workload-socket\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.313524 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.313497 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istiod-ca-cert\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.313653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.313634 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-data\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.313711 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.313701 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-token\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.415263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.415218 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-workload-socket\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.415433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.415269 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istiod-ca-cert\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.415433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.415316 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-data\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.415433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.415372 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-token\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.415433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.415402 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-credential-socket\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.415715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.415433 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-envoy\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.415715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.415481 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2phrl\" (UniqueName: \"kubernetes.io/projected/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-kube-api-access-2phrl\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.415842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.415707 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-workload-socket\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.415842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.415731 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-data\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.415953 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.415865 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-podinfo\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.415953 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.415906 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-credential-socket\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.416055 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.416021 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-workload-certs\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.416131 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.416110 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istiod-ca-cert\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.416349 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.416332 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-workload-certs\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.417901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.417875 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-envoy\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.418316 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.418299 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-podinfo\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.423649 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.423626 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-token\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.424023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.423998 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2phrl\" (UniqueName: \"kubernetes.io/projected/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-kube-api-access-2phrl\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.575190 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.575096 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:34.873297 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.873253 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5"] Nov 03 20:13:34.873749 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:13:34.873407 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf854add6_2ad9_4b27_a0a2_72bf8f2d6fb9.slice/crio-8a3caeb2d8c691a666b4129ea30ee64e0ad56af4b568a8b477dd93a802302c68 WatchSource:0}: Error finding container 8a3caeb2d8c691a666b4129ea30ee64e0ad56af4b568a8b477dd93a802302c68: Status 404 returned error can't find the container with id 8a3caeb2d8c691a666b4129ea30ee64e0ad56af4b568a8b477dd93a802302c68 Nov 03 20:13:34.877022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.876988 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:13:34.877112 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.877059 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:13:34.877112 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:34.877088 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:13:35.116621 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:13:35.116519 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:13:35.316223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:35.316120 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" event={"ID":"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9","Type":"ContainerStarted","Data":"6c56f4071ac822f088f3b10a8d07bb200c9b38b2914cbaf1d497f42b49b2078c"} Nov 03 20:13:35.316223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:35.316200 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" event={"ID":"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9","Type":"ContainerStarted","Data":"8a3caeb2d8c691a666b4129ea30ee64e0ad56af4b568a8b477dd93a802302c68"} Nov 03 20:13:35.342234 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:35.342172 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" podStartSLOduration=1.342153997 podStartE2EDuration="1.342153997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:35.338266505 +0000 UTC m=+2045.300644943" watchObservedRunningTime="2026-11-03 20:13:35.342153997 +0000 UTC m=+2045.304532436" Nov 03 20:13:35.575962 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:35.575880 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:35.577469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:35.577438 5111 patch_prober.go:29] interesting pod/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:13:35.577645 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:35.577484 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" podUID="f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:13:36.613487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:36.609128 5111 patch_prober.go:29] interesting pod/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:13:36.613487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:36.609195 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" podUID="f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:13:37.035551 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:37.033737 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s"] Nov 03 20:13:37.038946 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:37.038915 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e383a36-f7j6s"] Nov 03 20:13:37.093462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:37.093433 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" Nov 03 20:13:37.093875 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:37.093855 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" Nov 03 20:13:37.580378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:37.580234 5111 patch_prober.go:29] interesting pod/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:13:37.580378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:37.580315 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" podUID="f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:13:38.191108 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:38.191073 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="865c1010-f54c-4b3d-8dab-68f01d041930" path="/var/lib/kubelet/pods/865c1010-f54c-4b3d-8dab-68f01d041930/volumes" Nov 03 20:13:38.575951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:38.575873 5111 patch_prober.go:29] interesting pod/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:13:38.576190 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:38.576159 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" podUID="f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:13:39.588936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:39.588397 5111 patch_prober.go:29] interesting pod/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:13:39.588936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:39.588473 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" podUID="f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:13:40.602240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:40.601743 5111 patch_prober.go:29] interesting pod/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:13:40.602240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:40.601829 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" podUID="f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:13:41.576030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:41.575987 5111 patch_prober.go:29] interesting pod/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:13:41.576221 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:41.576048 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" podUID="f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:13:41.935601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:41.935565 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw"] Nov 03 20:13:41.944077 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:41.944050 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw"] Nov 03 20:13:41.944276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:41.944182 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:41.955880 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:41.955848 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9b1aed-openshift-default-dockercfg-9brqf\"" Nov 03 20:13:42.081291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.081252 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e140c17f-f261-4a57-938d-7199ddcce0a3-istiod-ca-cert\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.081478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.081369 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-workload-certs\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.081478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.081394 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-credential-socket\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.081634 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.081502 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-podinfo\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.081634 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.081617 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-data\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.081713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.081641 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-envoy\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.081791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.081779 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-token\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.081829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.081821 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-workload-socket\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.081917 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.081904 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djjvb\" (UniqueName: \"kubernetes.io/projected/e140c17f-f261-4a57-938d-7199ddcce0a3-kube-api-access-djjvb\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.183786 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.183742 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-podinfo\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.184003 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.183843 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-data\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.184003 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.183876 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-envoy\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.184003 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.183992 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-token\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.184179 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.184042 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-workload-socket\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.184179 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.184098 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djjvb\" (UniqueName: \"kubernetes.io/projected/e140c17f-f261-4a57-938d-7199ddcce0a3-kube-api-access-djjvb\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.184293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.184214 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e140c17f-f261-4a57-938d-7199ddcce0a3-istiod-ca-cert\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.184293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.184282 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-workload-certs\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.184403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.184316 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-credential-socket\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.185213 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.185111 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-credential-socket\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.185341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.185247 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-data\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.197616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.186025 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e140c17f-f261-4a57-938d-7199ddcce0a3-istiod-ca-cert\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.197616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.189252 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-envoy\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.201551 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.198083 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-workload-socket\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.201551 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.199934 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-workload-certs\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.230378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.230348 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-podinfo\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.245665 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.245634 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-token\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.267800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.267767 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djjvb\" (UniqueName: \"kubernetes.io/projected/e140c17f-f261-4a57-938d-7199ddcce0a3-kube-api-access-djjvb\") pod \"e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.277027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.276977 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:42.575559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.575463 5111 patch_prober.go:29] interesting pod/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:13:42.575559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.575509 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" podUID="f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:13:42.896363 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.896333 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw"] Nov 03 20:13:42.896812 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:13:42.896733 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode140c17f_f261_4a57_938d_7199ddcce0a3.slice/crio-1f6e008791d37027a97a71d1ad58ff42448601c048470953cb5093eae15d04b3 WatchSource:0}: Error finding container 1f6e008791d37027a97a71d1ad58ff42448601c048470953cb5093eae15d04b3: Status 404 returned error can't find the container with id 1f6e008791d37027a97a71d1ad58ff42448601c048470953cb5093eae15d04b3 Nov 03 20:13:42.900137 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.900099 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:13:42.900242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.900175 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:13:42.900242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:42.900208 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:13:43.582315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:43.582278 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:43.582868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:43.582815 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:43.583465 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:43.583443 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:13:43.700480 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:43.700437 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" event={"ID":"e140c17f-f261-4a57-938d-7199ddcce0a3","Type":"ContainerStarted","Data":"437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40"} Nov 03 20:13:43.700707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:43.700588 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" event={"ID":"e140c17f-f261-4a57-938d-7199ddcce0a3","Type":"ContainerStarted","Data":"1f6e008791d37027a97a71d1ad58ff42448601c048470953cb5093eae15d04b3"} Nov 03 20:13:43.728119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:43.728060 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" podStartSLOduration=2.728042582 podStartE2EDuration="2.728042582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:43.724205471 +0000 UTC m=+2053.686583908" watchObservedRunningTime="2026-11-03 20:13:43.728042582 +0000 UTC m=+2053.690421020" Nov 03 20:13:44.278332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:44.278297 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:44.285104 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:44.285076 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:44.711734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:44.711694 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:44.716949 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:44.716919 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:13:56.711070 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:56.711022 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r"] Nov 03 20:13:56.716954 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:56.716925 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-caa01a-nqf8r"] Nov 03 20:13:58.188741 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:58.188707 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617" path="/var/lib/kubelet/pods/2a2de4d0-c6bd-4cb8-9996-5ba8ece1d617/volumes" Nov 03 20:13:59.534017 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:59.533980 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8"] Nov 03 20:13:59.540164 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:59.540139 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" Nov 03 20:13:59.544770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:59.544747 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-22a567\"" Nov 03 20:13:59.544914 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:59.544900 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-22a567-dockercfg-cmljz\"" Nov 03 20:13:59.547333 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:59.547308 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8"] Nov 03 20:13:59.582324 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:59.582288 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af7b24bc-adce-4c7a-b2be-5844a4d44675-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8\" (UID: \"af7b24bc-adce-4c7a-b2be-5844a4d44675\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" Nov 03 20:13:59.582718 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:59.582689 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-876gk\" (UniqueName: \"kubernetes.io/projected/af7b24bc-adce-4c7a-b2be-5844a4d44675-kube-api-access-876gk\") pod \"payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8\" (UID: \"af7b24bc-adce-4c7a-b2be-5844a4d44675\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" Nov 03 20:13:59.772560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:59.771466 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af7b24bc-adce-4c7a-b2be-5844a4d44675-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8\" (UID: \"af7b24bc-adce-4c7a-b2be-5844a4d44675\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" Nov 03 20:13:59.772560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:59.771980 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-876gk\" (UniqueName: \"kubernetes.io/projected/af7b24bc-adce-4c7a-b2be-5844a4d44675-kube-api-access-876gk\") pod \"payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8\" (UID: \"af7b24bc-adce-4c7a-b2be-5844a4d44675\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" Nov 03 20:13:59.774856 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:59.773363 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af7b24bc-adce-4c7a-b2be-5844a4d44675-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8\" (UID: \"af7b24bc-adce-4c7a-b2be-5844a4d44675\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" Nov 03 20:13:59.900037 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:13:59.894001 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-876gk\" (UniqueName: \"kubernetes.io/projected/af7b24bc-adce-4c7a-b2be-5844a4d44675-kube-api-access-876gk\") pod \"payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8\" (UID: \"af7b24bc-adce-4c7a-b2be-5844a4d44675\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" Nov 03 20:14:00.159846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:00.159759 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" Nov 03 20:14:00.569781 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:14:00.569749 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf7b24bc_adce_4c7a_b2be_5844a4d44675.slice/crio-f162fea08b5473e83ff73dd3e2c7c61bd976ce00190617df5a87df727d805d47 WatchSource:0}: Error finding container f162fea08b5473e83ff73dd3e2c7c61bd976ce00190617df5a87df727d805d47: Status 404 returned error can't find the container with id f162fea08b5473e83ff73dd3e2c7c61bd976ce00190617df5a87df727d805d47 Nov 03 20:14:00.570774 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:00.570744 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8"] Nov 03 20:14:01.128395 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:01.128357 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" event={"ID":"af7b24bc-adce-4c7a-b2be-5844a4d44675","Type":"ContainerStarted","Data":"0002aa5a90ea141e32e45508480c90a1fb37a048cebf58148e9438179853dbb3"} Nov 03 20:14:01.128627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:01.128416 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" event={"ID":"af7b24bc-adce-4c7a-b2be-5844a4d44675","Type":"ContainerStarted","Data":"f162fea08b5473e83ff73dd3e2c7c61bd976ce00190617df5a87df727d805d47"} Nov 03 20:14:01.128627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:01.128461 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" Nov 03 20:14:01.151389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:01.151332 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" podStartSLOduration=2.15131882 podStartE2EDuration="2.15131882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:01.14742897 +0000 UTC m=+2071.109807444" watchObservedRunningTime="2026-11-03 20:14:01.15131882 +0000 UTC m=+2071.113697256" Nov 03 20:14:05.062688 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:14:05.062652 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:14:05.317807 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:05.317726 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x"] Nov 03 20:14:05.318031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:05.317955 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" podUID="e837c3cb-1bf5-4774-a9e1-a378f67048c1" containerName="istio-proxy" containerID="cri-o://4001bd6d523ac51f0c372783bc1b7c4ff42926e3d5277235f9487ba4cdb2d82b" gracePeriod=30 Nov 03 20:14:11.499841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.499776 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:14:11.503371 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.503340 5111 generic.go:356] "Generic (PLEG): container finished" podID="e837c3cb-1bf5-4774-a9e1-a378f67048c1" containerID="4001bd6d523ac51f0c372783bc1b7c4ff42926e3d5277235f9487ba4cdb2d82b" exitCode=0 Nov 03 20:14:11.503768 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.503742 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" event={"ID":"e837c3cb-1bf5-4774-a9e1-a378f67048c1","Type":"ContainerDied","Data":"4001bd6d523ac51f0c372783bc1b7c4ff42926e3d5277235f9487ba4cdb2d82b"} Nov 03 20:14:11.503981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.503945 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" event={"ID":"e837c3cb-1bf5-4774-a9e1-a378f67048c1","Type":"ContainerDied","Data":"fc0296637e69fd21f2421b9f85442e29020707722d378ea2de6136a361a2390a"} Nov 03 20:14:11.504144 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.504127 5111 scope.go:122] "RemoveContainer" containerID="4001bd6d523ac51f0c372783bc1b7c4ff42926e3d5277235f9487ba4cdb2d82b" Nov 03 20:14:11.626937 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.625976 5111 scope.go:122] "RemoveContainer" containerID="3466384fa52a7d9f63105b8efc6ff2eeb561039b9a79b7adb4866f27c0ecbd39" Nov 03 20:14:11.678960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.675219 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-workload-certs\") pod \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " Nov 03 20:14:11.678960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.675330 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e837c3cb-1bf5-4774-a9e1-a378f67048c1-kube-api-access-j8wwq\" (UniqueName: \"kubernetes.io/projected/e837c3cb-1bf5-4774-a9e1-a378f67048c1-kube-api-access-j8wwq\") pod \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " Nov 03 20:14:11.678960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.675437 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-data\") pod \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " Nov 03 20:14:11.689587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.682687 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-envoy\") pod \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " Nov 03 20:14:11.689587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.682831 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-credential-socket\") pod \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " Nov 03 20:14:11.691886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.691794 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-credential-socket" pod "e837c3cb-1bf5-4774-a9e1-a378f67048c1" (UID: "e837c3cb-1bf5-4774-a9e1-a378f67048c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:14:11.692151 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.692113 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-data" pod "e837c3cb-1bf5-4774-a9e1-a378f67048c1" (UID: "e837c3cb-1bf5-4774-a9e1-a378f67048c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:14:11.692605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.692579 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-workload-certs" pod "e837c3cb-1bf5-4774-a9e1-a378f67048c1" (UID: "e837c3cb-1bf5-4774-a9e1-a378f67048c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:14:11.696598 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.696569 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-workload-socket\") pod \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " Nov 03 20:14:11.696790 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.696772 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-podinfo\") pod \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " Nov 03 20:14:11.697015 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.696997 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istiod-ca-cert\") pod \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " Nov 03 20:14:11.697149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.697134 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-token\" (UniqueName: \"kubernetes.io/projected/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-token\") pod \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\" (UID: \"e837c3cb-1bf5-4774-a9e1-a378f67048c1\") " Nov 03 20:14:11.709283 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.698370 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:11.743946 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.735423 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:11.743946 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.735462 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:11.743946 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.707583 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istiod-ca-cert" pod "e837c3cb-1bf5-4774-a9e1-a378f67048c1" (UID: "e837c3cb-1bf5-4774-a9e1-a378f67048c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:14:11.743946 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.734782 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e837c3cb-1bf5-4774-a9e1-a378f67048c1-kube-api-access-j8wwq" pod "e837c3cb-1bf5-4774-a9e1-a378f67048c1" (UID: "e837c3cb-1bf5-4774-a9e1-a378f67048c1"). InnerVolumeSpecName "kube-api-access-j8wwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:11.753411 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.745268 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-workload-socket" pod "e837c3cb-1bf5-4774-a9e1-a378f67048c1" (UID: "e837c3cb-1bf5-4774-a9e1-a378f67048c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:14:11.774600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.770483 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc"] Nov 03 20:14:11.934820 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.893441 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:11.934820 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.893488 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:11.934820 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:11.893504 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8wwq\" (UniqueName: \"kubernetes.io/projected/e837c3cb-1bf5-4774-a9e1-a378f67048c1-kube-api-access-j8wwq\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:12.012556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.011091 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-podinfo" pod "e837c3cb-1bf5-4774-a9e1-a378f67048c1" (UID: "e837c3cb-1bf5-4774-a9e1-a378f67048c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:14:12.041560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.039128 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-envoy" pod "e837c3cb-1bf5-4774-a9e1-a378f67048c1" (UID: "e837c3cb-1bf5-4774-a9e1-a378f67048c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:14:12.075575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.068776 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-token" pod "e837c3cb-1bf5-4774-a9e1-a378f67048c1" (UID: "e837c3cb-1bf5-4774-a9e1-a378f67048c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:12.075575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.071808 5111 cpu_manager.go:414] "containerMap: removing container" podUID="e837c3cb-1bf5-4774-a9e1-a378f67048c1" containerName="istio-proxy" Nov 03 20:14:12.075575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.071848 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e837c3cb-1bf5-4774-a9e1-a378f67048c1" containerName="istio-proxy" Nov 03 20:14:12.075575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.071894 5111 cpu_manager.go:414] "containerMap: removing container" podUID="e837c3cb-1bf5-4774-a9e1-a378f67048c1" containerName="istio-proxy" Nov 03 20:14:12.075575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.071906 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e837c3cb-1bf5-4774-a9e1-a378f67048c1" containerName="istio-proxy" Nov 03 20:14:12.075575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.072213 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="e837c3cb-1bf5-4774-a9e1-a378f67048c1" containerName="istio-proxy" Nov 03 20:14:12.101556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.100862 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:12.101556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.100902 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:12.101556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.100914 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e837c3cb-1bf5-4774-a9e1-a378f67048c1-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:12.337001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.321123 5111 scope.go:122] "RemoveContainer" containerID="4001bd6d523ac51f0c372783bc1b7c4ff42926e3d5277235f9487ba4cdb2d82b" Nov 03 20:14:12.337001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.331336 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc"] Nov 03 20:14:12.337001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.331424 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" Nov 03 20:14:12.360213 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.358786 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.421157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.379575 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2c6c50-openshift-default-dockercfg-4phd2\"" Nov 03 20:14:12.421157 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:14:12.392186 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4001bd6d523ac51f0c372783bc1b7c4ff42926e3d5277235f9487ba4cdb2d82b\": container with ID starting with 4001bd6d523ac51f0c372783bc1b7c4ff42926e3d5277235f9487ba4cdb2d82b not found: ID does not exist" containerID="4001bd6d523ac51f0c372783bc1b7c4ff42926e3d5277235f9487ba4cdb2d82b" Nov 03 20:14:12.421157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.392326 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4001bd6d523ac51f0c372783bc1b7c4ff42926e3d5277235f9487ba4cdb2d82b"} err="failed to get container status \"4001bd6d523ac51f0c372783bc1b7c4ff42926e3d5277235f9487ba4cdb2d82b\": rpc error: code = NotFound desc = could not find container \"4001bd6d523ac51f0c372783bc1b7c4ff42926e3d5277235f9487ba4cdb2d82b\": container with ID starting with 4001bd6d523ac51f0c372783bc1b7c4ff42926e3d5277235f9487ba4cdb2d82b not found: ID does not exist" Nov 03 20:14:12.421157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.392359 5111 scope.go:122] "RemoveContainer" containerID="3466384fa52a7d9f63105b8efc6ff2eeb561039b9a79b7adb4866f27c0ecbd39" Nov 03 20:14:12.494845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.488877 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.494845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.489003 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.494845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.489048 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.494845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.489152 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.494845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.489181 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.494845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.489246 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.494845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.489314 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.494845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.489494 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glrmz\" (UniqueName: \"kubernetes.io/projected/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-kube-api-access-glrmz\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.494845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.489574 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.549076 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:14:12.548967 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3466384fa52a7d9f63105b8efc6ff2eeb561039b9a79b7adb4866f27c0ecbd39\": container with ID starting with 3466384fa52a7d9f63105b8efc6ff2eeb561039b9a79b7adb4866f27c0ecbd39 not found: ID does not exist" containerID="3466384fa52a7d9f63105b8efc6ff2eeb561039b9a79b7adb4866f27c0ecbd39" Nov 03 20:14:12.549076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.549014 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3466384fa52a7d9f63105b8efc6ff2eeb561039b9a79b7adb4866f27c0ecbd39"} err="failed to get container status \"3466384fa52a7d9f63105b8efc6ff2eeb561039b9a79b7adb4866f27c0ecbd39\": rpc error: code = NotFound desc = could not find container \"3466384fa52a7d9f63105b8efc6ff2eeb561039b9a79b7adb4866f27c0ecbd39\": container with ID starting with 3466384fa52a7d9f63105b8efc6ff2eeb561039b9a79b7adb4866f27c0ecbd39 not found: ID does not exist" Nov 03 20:14:12.567214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.567182 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x" Nov 03 20:14:12.625793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.625754 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.626004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.625972 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.626276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.626018 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.627036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.626518 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.627036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.626568 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.627426 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.627399 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.627507 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.627469 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.627680 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.627660 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glrmz\" (UniqueName: \"kubernetes.io/projected/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-kube-api-access-glrmz\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.627753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.627738 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.630066 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.630041 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x"] Nov 03 20:14:12.630802 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.630781 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.632838 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.632812 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.633748 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.633726 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.634116 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.634096 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.634498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.634473 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.637081 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.636404 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.665210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.665170 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glrmz\" (UniqueName: \"kubernetes.io/projected/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-kube-api-access-glrmz\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.674910 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.674880 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.677999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.677968 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-caa01a-openshift-default-7c546c7465-ndl4x"] Nov 03 20:14:12.692314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.692283 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:12.815604 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:12.814788 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:13.182252 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:13.182214 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc"] Nov 03 20:14:13.209057 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:14:13.209025 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58995fe_f4f0_48be_b3a0_03afb09e2ad3.slice/crio-14ad73a5c0272adf7cfd4e1148418af797f58b6f7e7b3ec6aca14bd9ae6ae4d9 WatchSource:0}: Error finding container 14ad73a5c0272adf7cfd4e1148418af797f58b6f7e7b3ec6aca14bd9ae6ae4d9: Status 404 returned error can't find the container with id 14ad73a5c0272adf7cfd4e1148418af797f58b6f7e7b3ec6aca14bd9ae6ae4d9 Nov 03 20:14:13.215129 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:13.215087 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:14:13.215436 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:13.215408 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:14:13.215641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:13.215617 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:14:13.678420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:13.678123 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" event={"ID":"c58995fe-f4f0-48be-b3a0-03afb09e2ad3","Type":"ContainerStarted","Data":"5cf1745f31dfecbb15e41cb1df7f5552ff9f2c672d0a4a0b36287f638eee13ec"} Nov 03 20:14:13.678420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:13.678326 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" event={"ID":"c58995fe-f4f0-48be-b3a0-03afb09e2ad3","Type":"ContainerStarted","Data":"14ad73a5c0272adf7cfd4e1148418af797f58b6f7e7b3ec6aca14bd9ae6ae4d9"} Nov 03 20:14:13.702337 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:13.702279 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" podStartSLOduration=7.702264046 podStartE2EDuration="7.702264046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:13.701853334 +0000 UTC m=+2083.664231773" watchObservedRunningTime="2026-11-03 20:14:13.702264046 +0000 UTC m=+2083.664642550" Nov 03 20:14:13.819492 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:13.819406 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:14.190136 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:14.190105 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e837c3cb-1bf5-4774-a9e1-a378f67048c1" path="/var/lib/kubelet/pods/e837c3cb-1bf5-4774-a9e1-a378f67048c1/volumes" Nov 03 20:14:14.821240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:14.821206 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:15.705513 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:15.705463 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:15.706334 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:15.706311 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:14:17.563658 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:17.563564 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf"] Nov 03 20:14:17.565257 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:17.565228 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="e837c3cb-1bf5-4774-a9e1-a378f67048c1" containerName="istio-proxy" Nov 03 20:14:17.571059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:17.571036 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" Nov 03 20:14:17.578267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:17.578230 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2c6c50\"" Nov 03 20:14:17.585263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:17.583277 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2c6c50-dockercfg-lvqfr\"" Nov 03 20:14:17.587469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:17.586213 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf"] Nov 03 20:14:17.719573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:17.718192 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k565z\" (UniqueName: \"kubernetes.io/projected/af81ede7-5819-41d1-94c8-0a297f814fb4-kube-api-access-k565z\") pod \"payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf\" (UID: \"af81ede7-5819-41d1-94c8-0a297f814fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" Nov 03 20:14:17.719573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:17.718282 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af81ede7-5819-41d1-94c8-0a297f814fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf\" (UID: \"af81ede7-5819-41d1-94c8-0a297f814fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" Nov 03 20:14:17.829664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:17.824589 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k565z\" (UniqueName: \"kubernetes.io/projected/af81ede7-5819-41d1-94c8-0a297f814fb4-kube-api-access-k565z\") pod \"payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf\" (UID: \"af81ede7-5819-41d1-94c8-0a297f814fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" Nov 03 20:14:17.829664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:17.824677 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af81ede7-5819-41d1-94c8-0a297f814fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf\" (UID: \"af81ede7-5819-41d1-94c8-0a297f814fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" Nov 03 20:14:17.829664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:17.825741 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af81ede7-5819-41d1-94c8-0a297f814fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf\" (UID: \"af81ede7-5819-41d1-94c8-0a297f814fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" Nov 03 20:14:18.062364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:18.062297 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k565z\" (UniqueName: \"kubernetes.io/projected/af81ede7-5819-41d1-94c8-0a297f814fb4-kube-api-access-k565z\") pod \"payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf\" (UID: \"af81ede7-5819-41d1-94c8-0a297f814fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" Nov 03 20:14:18.231853 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:18.231799 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" Nov 03 20:14:18.595334 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:18.595294 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf"] Nov 03 20:14:19.094063 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:19.094022 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" event={"ID":"af81ede7-5819-41d1-94c8-0a297f814fb4","Type":"ContainerStarted","Data":"22e0f0eb5b3fc5c7ca35c7749d2302dbffcb2ce7e2666ee91c4d166e87ee5bab"} Nov 03 20:14:19.094322 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:19.094082 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" event={"ID":"af81ede7-5819-41d1-94c8-0a297f814fb4","Type":"ContainerStarted","Data":"7fe6b3255f126490823fcd166070a20221a60fb20e4ef7d97c516f552e112806"} Nov 03 20:14:19.094322 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:19.094115 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" Nov 03 20:14:19.125499 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:19.125426 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" podStartSLOduration=2.125405977 podStartE2EDuration="2.125405977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:19.116296747 +0000 UTC m=+2089.078675186" watchObservedRunningTime="2026-11-03 20:14:19.125405977 +0000 UTC m=+2089.087784416" Nov 03 20:14:30.107688 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:30.107633 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" Nov 03 20:14:30.672168 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:30.672138 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:14:30.678382 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:30.678354 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:14:30.679303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:30.679275 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:14:30.685339 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:30.685311 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:14:33.725485 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:33.725449 5111 scope.go:122] "RemoveContainer" containerID="f7558f704537297532143c6030d2402c1c59a77f18cf2416e48c2fb83954a280" Nov 03 20:14:33.741695 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:33.741007 5111 scope.go:122] "RemoveContainer" containerID="17ce9e6e530e9f38e21bda707c2be5704ecba34792e5c26d95e4dab8491ca38c" Nov 03 20:14:33.754785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:33.754763 5111 scope.go:122] "RemoveContainer" containerID="e3e792cc40043536a8d9e353f33a87dd2190f3cb9e9ce615357b9556915659d5" Nov 03 20:14:35.066389 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:14:35.066350 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:14:53.503901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:53.503804 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz"] Nov 03 20:14:53.504401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:53.504080 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" podUID="1e1df657-2ab6-4ae7-80e5-0c6358becefd" containerName="istio-proxy" containerID="cri-o://98514336e723f7a5c0eac41318036eaa82e8c4bf2ba5ee7fb550ae57a93f240a" gracePeriod=30 Nov 03 20:14:54.435557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:54.426603 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:14:54.435557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:54.426704 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" podUID="1e1df657-2ab6-4ae7-80e5-0c6358becefd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:14:58.856780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:58.842476 5111 generic.go:356] "Generic (PLEG): container finished" podID="1e1df657-2ab6-4ae7-80e5-0c6358becefd" containerID="98514336e723f7a5c0eac41318036eaa82e8c4bf2ba5ee7fb550ae57a93f240a" exitCode=0 Nov 03 20:14:58.856780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:58.842526 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" event={"ID":"1e1df657-2ab6-4ae7-80e5-0c6358becefd","Type":"ContainerDied","Data":"98514336e723f7a5c0eac41318036eaa82e8c4bf2ba5ee7fb550ae57a93f240a"} Nov 03 20:14:58.856780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:58.842727 5111 scope.go:122] "RemoveContainer" containerID="25e2de8c87631905c805661111eadd55007460a31d6054d6fa01c651f13302f7" Nov 03 20:14:59.006737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.006706 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:14:59.135859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.135801 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf"] Nov 03 20:14:59.137364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.137333 5111 cpu_manager.go:414] "containerMap: removing container" podUID="1e1df657-2ab6-4ae7-80e5-0c6358becefd" containerName="istio-proxy" Nov 03 20:14:59.137364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.137366 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e1df657-2ab6-4ae7-80e5-0c6358becefd" containerName="istio-proxy" Nov 03 20:14:59.137608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.137385 5111 cpu_manager.go:414] "containerMap: removing container" podUID="1e1df657-2ab6-4ae7-80e5-0c6358becefd" containerName="istio-proxy" Nov 03 20:14:59.137608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.137395 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e1df657-2ab6-4ae7-80e5-0c6358becefd" containerName="istio-proxy" Nov 03 20:14:59.143785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.143758 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e1df657-2ab6-4ae7-80e5-0c6358becefd" containerName="istio-proxy" Nov 03 20:14:59.143901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.143795 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e1df657-2ab6-4ae7-80e5-0c6358becefd" containerName="istio-proxy" Nov 03 20:14:59.199298 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.199267 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf"] Nov 03 20:14:59.199683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.199662 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:14:59.214006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.213977 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-22a567-serving-cert\"" Nov 03 20:14:59.224829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.224781 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-credential-socket\") pod \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " Nov 03 20:14:59.224976 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.224841 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-envoy\") pod \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " Nov 03 20:14:59.224976 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.224938 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-workload-socket\") pod \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " Nov 03 20:14:59.225095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.225019 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-podinfo\") pod \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " Nov 03 20:14:59.225196 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.225173 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-token\" (UniqueName: \"kubernetes.io/projected/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-token\") pod \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " Nov 03 20:14:59.225266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.225248 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-data\") pod \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " Nov 03 20:14:59.231097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.225283 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e1df657-2ab6-4ae7-80e5-0c6358becefd-kube-api-access-ppbr4\" (UniqueName: \"kubernetes.io/projected/1e1df657-2ab6-4ae7-80e5-0c6358becefd-kube-api-access-ppbr4\") pod \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " Nov 03 20:14:59.231097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.225406 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istiod-ca-cert\") pod \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " Nov 03 20:14:59.231097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.225432 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-workload-certs\") pod \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\" (UID: \"1e1df657-2ab6-4ae7-80e5-0c6358becefd\") " Nov 03 20:14:59.231097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.225916 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-maas-api-tls\") pod \"maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf\" (UID: \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:14:59.231097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.226007 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmmcv\" (UniqueName: \"kubernetes.io/projected/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-kube-api-access-gmmcv\") pod \"maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf\" (UID: \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:14:59.231097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.226088 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-openshift-service-ca\") pod \"maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf\" (UID: \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:14:59.231097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.226132 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-metrics-certs\") pod \"maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf\" (UID: \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:14:59.231097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.226490 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-workload-socket" pod "1e1df657-2ab6-4ae7-80e5-0c6358becefd" (UID: "1e1df657-2ab6-4ae7-80e5-0c6358becefd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:14:59.231097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.227043 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-data" pod "1e1df657-2ab6-4ae7-80e5-0c6358becefd" (UID: "1e1df657-2ab6-4ae7-80e5-0c6358becefd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:14:59.231097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.227895 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istiod-ca-cert" pod "1e1df657-2ab6-4ae7-80e5-0c6358becefd" (UID: "1e1df657-2ab6-4ae7-80e5-0c6358becefd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:14:59.233523 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.232258 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-workload-certs" pod "1e1df657-2ab6-4ae7-80e5-0c6358becefd" (UID: "1e1df657-2ab6-4ae7-80e5-0c6358becefd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:14:59.233523 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.232994 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-credential-socket" pod "1e1df657-2ab6-4ae7-80e5-0c6358becefd" (UID: "1e1df657-2ab6-4ae7-80e5-0c6358becefd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:14:59.237918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.237880 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-podinfo" pod "1e1df657-2ab6-4ae7-80e5-0c6358becefd" (UID: "1e1df657-2ab6-4ae7-80e5-0c6358becefd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:14:59.271700 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.262458 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-envoy" pod "1e1df657-2ab6-4ae7-80e5-0c6358becefd" (UID: "1e1df657-2ab6-4ae7-80e5-0c6358becefd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:14:59.271700 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.262722 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e1df657-2ab6-4ae7-80e5-0c6358becefd-kube-api-access-ppbr4" pod "1e1df657-2ab6-4ae7-80e5-0c6358becefd" (UID: "1e1df657-2ab6-4ae7-80e5-0c6358becefd"). InnerVolumeSpecName "kube-api-access-ppbr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:59.271700 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.262814 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-token" pod "1e1df657-2ab6-4ae7-80e5-0c6358becefd" (UID: "1e1df657-2ab6-4ae7-80e5-0c6358becefd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:59.328224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.328185 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-maas-api-tls\") pod \"maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf\" (UID: \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:14:59.328428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.328364 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmmcv\" (UniqueName: \"kubernetes.io/projected/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-kube-api-access-gmmcv\") pod \"maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf\" (UID: \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:14:59.328510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.328490 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-openshift-service-ca\") pod \"maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf\" (UID: \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:14:59.328599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.328568 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-metrics-certs\") pod \"maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf\" (UID: \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:14:59.329061 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.328969 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:59.329061 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.328990 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:59.329061 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.329000 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:59.329061 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.329009 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:59.329061 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.329017 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:59.329061 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.329025 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:59.329061 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.329035 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppbr4\" (UniqueName: \"kubernetes.io/projected/1e1df657-2ab6-4ae7-80e5-0c6358becefd-kube-api-access-ppbr4\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:59.329061 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.329044 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e1df657-2ab6-4ae7-80e5-0c6358becefd-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:59.329061 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.329053 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e1df657-2ab6-4ae7-80e5-0c6358becefd-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:59.329444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.329294 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-openshift-service-ca\") pod \"maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf\" (UID: \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:14:59.331626 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.331599 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-maas-api-tls\") pod \"maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf\" (UID: \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:14:59.331915 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.331897 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-metrics-certs\") pod \"maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf\" (UID: \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:14:59.336319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.336299 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmmcv\" (UniqueName: \"kubernetes.io/projected/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-kube-api-access-gmmcv\") pod \"maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf\" (UID: \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:14:59.569905 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.569811 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:14:59.877569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.877523 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf"] Nov 03 20:14:59.878090 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:14:59.878068 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d3a0d1_0fd0_4c23_b448_bda0ceb3cd25.slice/crio-507fb9b90169de8522fa04cb2feff6bc60ee49c0bb9f8edeee23d7ce2327f766 WatchSource:0}: Error finding container 507fb9b90169de8522fa04cb2feff6bc60ee49c0bb9f8edeee23d7ce2327f766: Status 404 returned error can't find the container with id 507fb9b90169de8522fa04cb2feff6bc60ee49c0bb9f8edeee23d7ce2327f766 Nov 03 20:14:59.881506 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.881093 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" event={"ID":"1e1df657-2ab6-4ae7-80e5-0c6358becefd","Type":"ContainerDied","Data":"6ac563082718fbda412a6933d80060ffc5731b52c93966780e29cc3c0e139c22"} Nov 03 20:14:59.881506 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.881152 5111 scope.go:122] "RemoveContainer" containerID="98514336e723f7a5c0eac41318036eaa82e8c4bf2ba5ee7fb550ae57a93f240a" Nov 03 20:14:59.881506 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:14:59.881282 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz" Nov 03 20:15:00.008083 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:00.008048 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz"] Nov 03 20:15:00.013058 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:00.013030 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-d9gnz"] Nov 03 20:15:00.204059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:00.203968 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e1df657-2ab6-4ae7-80e5-0c6358becefd" path="/var/lib/kubelet/pods/1e1df657-2ab6-4ae7-80e5-0c6358becefd/volumes" Nov 03 20:15:00.914670 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:00.914628 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" event={"ID":"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25","Type":"ContainerStarted","Data":"0bbfc98f0956d085143594be6bd79313555cd2a6408094063391aace1a131767"} Nov 03 20:15:00.915304 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:00.914696 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" event={"ID":"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25","Type":"ContainerStarted","Data":"507fb9b90169de8522fa04cb2feff6bc60ee49c0bb9f8edeee23d7ce2327f766"} Nov 03 20:15:00.915304 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:00.914729 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:15:00.935662 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:00.935249 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" podStartSLOduration=40.309967132 podStartE2EDuration="40.935229235s" totalImagesPullingTime="625.262103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:00.932999809 +0000 UTC m=+2130.895378248" watchObservedRunningTime="2026-11-03 20:15:00.935229235 +0000 UTC m=+2130.897607675" Nov 03 20:15:05.278626 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:15:05.278517 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:15:06.934258 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:06.934229 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:15:14.712452 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:14.712411 5111 patch_prober.go:29] interesting pod/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:15:14.713004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:14.712486 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" podUID="e140c17f-f261-4a57-938d-7199ddcce0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:15:15.095723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:15.095636 5111 generic.go:356] "Generic (PLEG): container finished" podID="e140c17f-f261-4a57-938d-7199ddcce0a3" containerID="437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40" exitCode=0 Nov 03 20:15:15.095723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:15.095697 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" event={"ID":"e140c17f-f261-4a57-938d-7199ddcce0a3","Type":"ContainerDied","Data":"437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40"} Nov 03 20:15:15.096269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:15.096243 5111 scope.go:122] "RemoveContainer" containerID="437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40" Nov 03 20:15:15.098356 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:15.098324 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:15:15.098459 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:15.098397 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:15:15.098459 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:15.098435 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:15:16.109456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:16.109424 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" event={"ID":"e140c17f-f261-4a57-938d-7199ddcce0a3","Type":"ContainerStarted","Data":"0edd1061ffc6c63e0cb8ed0b28562f31af788220f8e2505de1f96d5e2e1a7ed8"} Nov 03 20:15:16.277439 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:16.277406 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:15:16.277658 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:16.277632 5111 patch_prober.go:29] interesting pod/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:15:16.277725 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:16.277704 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" podUID="e140c17f-f261-4a57-938d-7199ddcce0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:15:17.284555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:17.283925 5111 patch_prober.go:29] interesting pod/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:15:17.284555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:17.283992 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" podUID="e140c17f-f261-4a57-938d-7199ddcce0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:15:18.278595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:18.278549 5111 patch_prober.go:29] interesting pod/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:15:18.278818 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:18.278650 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" podUID="e140c17f-f261-4a57-938d-7199ddcce0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:15:19.284112 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:19.281989 5111 patch_prober.go:29] interesting pod/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:15:19.284112 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:19.282081 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" podUID="e140c17f-f261-4a57-938d-7199ddcce0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:15:20.278603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:20.278556 5111 patch_prober.go:29] interesting pod/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:15:20.279028 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:20.278996 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" podUID="e140c17f-f261-4a57-938d-7199ddcce0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:15:21.277664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:21.277626 5111 patch_prober.go:29] interesting pod/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:15:21.279208 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:21.277701 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" podUID="e140c17f-f261-4a57-938d-7199ddcce0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:15:22.375315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:22.375278 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:15:22.375901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:22.375854 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:15:22.376746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:22.376721 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:15:31.746730 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:31.746693 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:31.747275 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:31.746764 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" podUID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:33.811456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:33.811417 5111 scope.go:122] "RemoveContainer" containerID="c84d9bb6f0a029df16cb5679ed1d6f7e77b9d011e492b5cb52d55a239e8b85ec" Nov 03 20:15:34.145396 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:34.125492 5111 scope.go:122] "RemoveContainer" containerID="cc02f0bdf15cc69694990417bb01faf67841276bd2e9e1201e4be71602e62d8f" Nov 03 20:15:35.311063 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:15:35.311021 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:15:37.025090 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:37.025050 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x"] Nov 03 20:15:37.065858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:37.030603 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e1cb6c-6s62x"] Nov 03 20:15:39.192759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:39.192722 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3484cb6-490f-409b-9a86-0ab06f1a4438" path="/var/lib/kubelet/pods/f3484cb6-490f-409b-9a86-0ab06f1a4438/volumes" Nov 03 20:15:39.285795 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:15:39.285763 5111 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.111s" Nov 03 20:15:46.692504 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:46.692378 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": read tcp 10.132.0.2:42584->10.132.0.209:15021: read: connection reset by peer" start-of-body= Nov 03 20:15:46.692504 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:46.692445 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" podUID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": read tcp 10.132.0.2:42584->10.132.0.209:15021: read: connection reset by peer" Nov 03 20:15:47.195105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:47.195073 5111 generic.go:356] "Generic (PLEG): container finished" podID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" containerID="5cf1745f31dfecbb15e41cb1df7f5552ff9f2c672d0a4a0b36287f638eee13ec" exitCode=0 Nov 03 20:15:47.195333 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:47.195149 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" event={"ID":"c58995fe-f4f0-48be-b3a0-03afb09e2ad3","Type":"ContainerDied","Data":"5cf1745f31dfecbb15e41cb1df7f5552ff9f2c672d0a4a0b36287f638eee13ec"} Nov 03 20:15:47.195746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:47.195726 5111 scope.go:122] "RemoveContainer" containerID="5cf1745f31dfecbb15e41cb1df7f5552ff9f2c672d0a4a0b36287f638eee13ec" Nov 03 20:15:47.198208 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:47.198179 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:15:47.198286 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:47.198255 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:15:47.198325 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:47.198287 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:15:48.205257 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:48.205184 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" event={"ID":"c58995fe-f4f0-48be-b3a0-03afb09e2ad3","Type":"ContainerStarted","Data":"00327e7941e5036b29f28150fa92dd0818abad17290fe7f20147fe7926729a0c"} Nov 03 20:15:48.815317 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:48.815279 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:15:48.815571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:48.815458 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:15:48.815571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:48.815509 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" podUID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 03 20:15:49.815765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:49.815730 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:15:49.816180 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:49.815853 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" podUID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 03 20:15:50.061635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:50.061590 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5"] Nov 03 20:15:50.094771 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:50.092988 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d78eb9a-qmcj5"] Nov 03 20:15:50.197940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:50.197905 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d60aeef-cb74-40e4-b1aa-42fd6ee7962c" path="/var/lib/kubelet/pods/6d60aeef-cb74-40e4-b1aa-42fd6ee7962c/volumes" Nov 03 20:15:50.815585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:50.815548 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:15:50.816388 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:50.815601 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" podUID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 03 20:15:51.815102 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:51.815068 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:15:51.815371 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:51.815117 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" podUID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 03 20:15:52.815143 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:52.815105 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:15:52.815602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:52.815165 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" podUID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 03 20:15:53.840589 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:53.840542 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:15:53.841120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:53.841045 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:15:53.842663 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:15:53.842638 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:16:05.065476 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:16:05.065438 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:16:34.437806 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:16:34.437775 5111 scope.go:122] "RemoveContainer" containerID="965c7fa831155105c66e78ac75f683aee3773a5381c46131432b83e0632edb49" Nov 03 20:16:34.450763 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:16:34.450734 5111 scope.go:122] "RemoveContainer" containerID="e556da01e315113b25cdbdac5e2a273fc45ddac0b3d36fea0aa03929cf33043f" Nov 03 20:16:34.465118 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:16:34.465080 5111 scope.go:122] "RemoveContainer" containerID="e7fd613f0c302b7fffabe6f0df8d24c704f83a581b4f110f5512a573aa568350" Nov 03 20:16:34.477706 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:16:34.476761 5111 scope.go:122] "RemoveContainer" containerID="12917f41f64b6e44fa4671cd1f087fdb23955d3c1324ad8ae3a580a7735bbb0a" Nov 03 20:16:35.061716 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:16:35.061680 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:16:41.929561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:16:41.928618 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:41.982557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:16:41.981614 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:05.065584 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:17:05.065525 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:17:34.628560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:34.621959 5111 scope.go:122] "RemoveContainer" containerID="79dbc5eda539c25bd3150a9a767896bf663a98b9f623f6172af4f0ddc4fd2f54" Nov 03 20:17:34.717772 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:34.707266 5111 scope.go:122] "RemoveContainer" containerID="830ab6ffee97e1ac7d2f2bc964541e3406de7a4440eb27b671f0b2ec348c725a" Nov 03 20:17:34.746895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:34.746727 5111 scope.go:122] "RemoveContainer" containerID="78a6846a729f3b96c18b1bb9ae407b86170bba31fb01306ed66f64f26608745e" Nov 03 20:17:35.126892 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:17:35.126854 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:17:53.999904 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:53.999816 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk"] Nov 03 20:17:54.001295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:54.000050 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" podUID="48a90be1-f894-4f95-8ccc-f8d2bd2d929d" containerName="istio-proxy" containerID="cri-o://3da3f0cbd0993adee23c0349751ff21b3ea92ed64537be82304bfeb73f7be019" gracePeriod=30 Nov 03 20:17:54.302641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:54.302461 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv"] Nov 03 20:17:54.302865 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:54.302789 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" podUID="7310a5cf-4706-445f-8c50-95e2f57c1ee2" containerName="istio-proxy" containerID="cri-o://ff970f9879c31eda0bd975b875951654bf74d5231a34d00487b28714979e50fb" gracePeriod=30 Nov 03 20:17:59.437020 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.436965 5111 generic.go:356] "Generic (PLEG): container finished" podID="48a90be1-f894-4f95-8ccc-f8d2bd2d929d" containerID="3da3f0cbd0993adee23c0349751ff21b3ea92ed64537be82304bfeb73f7be019" exitCode=0 Nov 03 20:17:59.438609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.437031 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" event={"ID":"48a90be1-f894-4f95-8ccc-f8d2bd2d929d","Type":"ContainerDied","Data":"3da3f0cbd0993adee23c0349751ff21b3ea92ed64537be82304bfeb73f7be019"} Nov 03 20:17:59.545195 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.540619 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:17:59.633746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.630907 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f"] Nov 03 20:17:59.635866 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.635818 5111 cpu_manager.go:414] "containerMap: removing container" podUID="48a90be1-f894-4f95-8ccc-f8d2bd2d929d" containerName="istio-proxy" Nov 03 20:17:59.635866 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.635849 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a90be1-f894-4f95-8ccc-f8d2bd2d929d" containerName="istio-proxy" Nov 03 20:17:59.637000 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.636488 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a90be1-f894-4f95-8ccc-f8d2bd2d929d" containerName="istio-proxy" Nov 03 20:17:59.642545 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.642507 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.648134 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.648106 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-credential-socket\") pod \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " Nov 03 20:17:59.648283 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.648239 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-token\" (UniqueName: \"kubernetes.io/projected/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-token\") pod \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " Nov 03 20:17:59.648350 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.648278 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-kube-api-access-smfr2\" (UniqueName: \"kubernetes.io/projected/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-kube-api-access-smfr2\") pod \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " Nov 03 20:17:59.648350 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.648323 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-podinfo\") pod \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " Nov 03 20:17:59.649246 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.648372 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-envoy\") pod \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " Nov 03 20:17:59.649246 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.648417 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-workload-certs\") pod \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " Nov 03 20:17:59.649246 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.648452 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f"] Nov 03 20:17:59.649246 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.648614 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-data\") pod \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " Nov 03 20:17:59.649246 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.648787 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istiod-ca-cert\") pod \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " Nov 03 20:17:59.649246 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.648863 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-workload-socket\") pod \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\" (UID: \"48a90be1-f894-4f95-8ccc-f8d2bd2d929d\") " Nov 03 20:17:59.652373 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.649911 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-workload-certs" pod "48a90be1-f894-4f95-8ccc-f8d2bd2d929d" (UID: "48a90be1-f894-4f95-8ccc-f8d2bd2d929d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:59.652373 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.649965 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-credential-socket" pod "48a90be1-f894-4f95-8ccc-f8d2bd2d929d" (UID: "48a90be1-f894-4f95-8ccc-f8d2bd2d929d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:59.652373 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.650605 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istiod-ca-cert" pod "48a90be1-f894-4f95-8ccc-f8d2bd2d929d" (UID: "48a90be1-f894-4f95-8ccc-f8d2bd2d929d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:59.653642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.652849 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-workload-socket" pod "48a90be1-f894-4f95-8ccc-f8d2bd2d929d" (UID: "48a90be1-f894-4f95-8ccc-f8d2bd2d929d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:59.657697 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.657658 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-data" pod "48a90be1-f894-4f95-8ccc-f8d2bd2d929d" (UID: "48a90be1-f894-4f95-8ccc-f8d2bd2d929d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:59.658747 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.658727 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.658747 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.658747 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.658922 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.658756 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.658922 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.658765 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.660251 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.660228 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-envoy" pod "48a90be1-f894-4f95-8ccc-f8d2bd2d929d" (UID: "48a90be1-f894-4f95-8ccc-f8d2bd2d929d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:59.660936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.660287 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-kube-api-access-smfr2" pod "48a90be1-f894-4f95-8ccc-f8d2bd2d929d" (UID: "48a90be1-f894-4f95-8ccc-f8d2bd2d929d"). InnerVolumeSpecName "kube-api-access-smfr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:59.660936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.660310 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-token" pod "48a90be1-f894-4f95-8ccc-f8d2bd2d929d" (UID: "48a90be1-f894-4f95-8ccc-f8d2bd2d929d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:59.662160 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.662133 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-podinfo" pod "48a90be1-f894-4f95-8ccc-f8d2bd2d929d" (UID: "48a90be1-f894-4f95-8ccc-f8d2bd2d929d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:17:59.711437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.711402 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:17:59.752587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.752549 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j"] Nov 03 20:17:59.754499 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.754467 5111 cpu_manager.go:414] "containerMap: removing container" podUID="7310a5cf-4706-445f-8c50-95e2f57c1ee2" containerName="istio-proxy" Nov 03 20:17:59.754691 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.754624 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7310a5cf-4706-445f-8c50-95e2f57c1ee2" containerName="istio-proxy" Nov 03 20:17:59.754971 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.754944 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="7310a5cf-4706-445f-8c50-95e2f57c1ee2" containerName="istio-proxy" Nov 03 20:17:59.760456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.760393 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.760627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.760469 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.760627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.760517 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.760627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.760585 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.760862 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.760827 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.760921 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.760891 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.761717 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.761681 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c48c95-5c93-4887-86bb-8e2b76c120bf-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.761856 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.761744 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x7cl\" (UniqueName: \"kubernetes.io/projected/30c48c95-5c93-4887-86bb-8e2b76c120bf-kube-api-access-8x7cl\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.761856 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.761789 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.762046 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.762027 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.762220 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.762200 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smfr2\" (UniqueName: \"kubernetes.io/projected/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-kube-api-access-smfr2\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.762366 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.762350 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.762499 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.762475 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.762691 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.762673 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a90be1-f894-4f95-8ccc-f8d2bd2d929d-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.766366 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.766317 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j"] Nov 03 20:17:59.766673 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.766454 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.864079 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.864036 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-credential-socket\") pod \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " Nov 03 20:17:59.864262 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.864118 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-podinfo\") pod \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " Nov 03 20:17:59.864363 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.864346 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7310a5cf-4706-445f-8c50-95e2f57c1ee2-kube-api-access-pthhg\" (UniqueName: \"kubernetes.io/projected/7310a5cf-4706-445f-8c50-95e2f57c1ee2-kube-api-access-pthhg\") pod \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.864385 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-workload-socket\") pod \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.864403 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-credential-socket" pod "7310a5cf-4706-445f-8c50-95e2f57c1ee2" (UID: "7310a5cf-4706-445f-8c50-95e2f57c1ee2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.864442 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-workload-certs\") pod \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.864523 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-token\" (UniqueName: \"kubernetes.io/projected/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-token\") pod \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.864590 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istiod-ca-cert\") pod \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.864629 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-data\") pod \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.864720 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-envoy\") pod \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\" (UID: \"7310a5cf-4706-445f-8c50-95e2f57c1ee2\") " Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.864754 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-workload-socket" pod "7310a5cf-4706-445f-8c50-95e2f57c1ee2" (UID: "7310a5cf-4706-445f-8c50-95e2f57c1ee2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.864781 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-workload-certs" pod "7310a5cf-4706-445f-8c50-95e2f57c1ee2" (UID: "7310a5cf-4706-445f-8c50-95e2f57c1ee2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.864963 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.864966 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-data" pod "7310a5cf-4706-445f-8c50-95e2f57c1ee2" (UID: "7310a5cf-4706-445f-8c50-95e2f57c1ee2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865011 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istiod-ca-cert" pod "7310a5cf-4706-445f-8c50-95e2f57c1ee2" (UID: "7310a5cf-4706-445f-8c50-95e2f57c1ee2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865029 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865203 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865263 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865303 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865336 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/226bf833-ba6d-4c44-91bd-015817c60b68-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865417 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865450 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865481 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.865720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865602 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226bf833-ba6d-4c44-91bd-015817c60b68-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865742 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226bf833-ba6d-4c44-91bd-015817c60b68-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865788 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865812 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865847 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72rxl\" (UniqueName: \"kubernetes.io/projected/226bf833-ba6d-4c44-91bd-015817c60b68-kube-api-access-72rxl\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865907 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865937 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.865983 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.866009 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c48c95-5c93-4887-86bb-8e2b76c120bf-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.866047 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x7cl\" (UniqueName: \"kubernetes.io/projected/30c48c95-5c93-4887-86bb-8e2b76c120bf-kube-api-access-8x7cl\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.866095 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.866205 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.866272 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.866986 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-podinfo" pod "7310a5cf-4706-445f-8c50-95e2f57c1ee2" (UID: "7310a5cf-4706-445f-8c50-95e2f57c1ee2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.867055 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-token" pod "7310a5cf-4706-445f-8c50-95e2f57c1ee2" (UID: "7310a5cf-4706-445f-8c50-95e2f57c1ee2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.867080 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c48c95-5c93-4887-86bb-8e2b76c120bf-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.867198 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.867218 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.867235 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.867248 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.867263 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.867276 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.867327 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-envoy" pod "7310a5cf-4706-445f-8c50-95e2f57c1ee2" (UID: "7310a5cf-4706-445f-8c50-95e2f57c1ee2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.868310 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7310a5cf-4706-445f-8c50-95e2f57c1ee2-kube-api-access-pthhg" pod "7310a5cf-4706-445f-8c50-95e2f57c1ee2" (UID: "7310a5cf-4706-445f-8c50-95e2f57c1ee2"). InnerVolumeSpecName "kube-api-access-pthhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.869671 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.870837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.869951 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.875458 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.875435 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.875807 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.875789 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x7cl\" (UniqueName: \"kubernetes.io/projected/30c48c95-5c93-4887-86bb-8e2b76c120bf-kube-api-access-8x7cl\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:17:59.976434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.976321 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.976434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.976406 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/226bf833-ba6d-4c44-91bd-015817c60b68-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.976797 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.976462 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.976797 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.976603 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226bf833-ba6d-4c44-91bd-015817c60b68-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.976797 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.976634 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226bf833-ba6d-4c44-91bd-015817c60b68-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.976797 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.976710 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72rxl\" (UniqueName: \"kubernetes.io/projected/226bf833-ba6d-4c44-91bd-015817c60b68-kube-api-access-72rxl\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.977026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.976862 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.977026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.976979 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.977026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.977006 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.977273 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.977233 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pthhg\" (UniqueName: \"kubernetes.io/projected/7310a5cf-4706-445f-8c50-95e2f57c1ee2-kube-api-access-pthhg\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.977273 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.977257 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.977388 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.977275 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7310a5cf-4706-445f-8c50-95e2f57c1ee2-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:59.977734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.977710 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.977950 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.977934 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.981939 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.981904 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.982198 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.982171 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.982854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.982826 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226bf833-ba6d-4c44-91bd-015817c60b68-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.983828 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.982847 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.983828 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.983247 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226bf833-ba6d-4c44-91bd-015817c60b68-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.986105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.986083 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/226bf833-ba6d-4c44-91bd-015817c60b68-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:17:59.987517 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:17:59.987497 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72rxl\" (UniqueName: \"kubernetes.io/projected/226bf833-ba6d-4c44-91bd-015817c60b68-kube-api-access-72rxl\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:18:00.010105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.010049 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:18:00.090095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.090053 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:18:00.337173 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.337139 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f"] Nov 03 20:18:00.342898 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:18:00.342849 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c48c95_5c93_4887_86bb_8e2b76c120bf.slice/crio-281d4a37f07b025f0e770cdd92fedd7d98fe780fcfad967340d419454b9dcfa2 WatchSource:0}: Error finding container 281d4a37f07b025f0e770cdd92fedd7d98fe780fcfad967340d419454b9dcfa2: Status 404 returned error can't find the container with id 281d4a37f07b025f0e770cdd92fedd7d98fe780fcfad967340d419454b9dcfa2 Nov 03 20:18:00.347194 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.347148 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:18:00.348395 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.347251 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:18:00.348395 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.347295 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:18:00.429369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.429335 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j"] Nov 03 20:18:00.431413 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:18:00.431389 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226bf833_ba6d_4c44_91bd_015817c60b68.slice/crio-c1fc201a572eb5617d3528d20b5643fb3e469ddeda8104b01205a9516000632b WatchSource:0}: Error finding container c1fc201a572eb5617d3528d20b5643fb3e469ddeda8104b01205a9516000632b: Status 404 returned error can't find the container with id c1fc201a572eb5617d3528d20b5643fb3e469ddeda8104b01205a9516000632b Nov 03 20:18:00.435233 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.435188 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:18:00.435357 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.435287 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:18:00.435357 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.435337 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:18:00.448689 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.448658 5111 generic.go:356] "Generic (PLEG): container finished" podID="7310a5cf-4706-445f-8c50-95e2f57c1ee2" containerID="ff970f9879c31eda0bd975b875951654bf74d5231a34d00487b28714979e50fb" exitCode=0 Nov 03 20:18:00.449207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.448721 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" Nov 03 20:18:00.449207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.448746 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" event={"ID":"7310a5cf-4706-445f-8c50-95e2f57c1ee2","Type":"ContainerDied","Data":"ff970f9879c31eda0bd975b875951654bf74d5231a34d00487b28714979e50fb"} Nov 03 20:18:00.449207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.448843 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv" event={"ID":"7310a5cf-4706-445f-8c50-95e2f57c1ee2","Type":"ContainerDied","Data":"4e1c7324b7c82a55c0c262a128898780eeb82f9cd9ed91ffece3d0f9b10d8bfb"} Nov 03 20:18:00.449207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.448880 5111 scope.go:122] "RemoveContainer" containerID="ff970f9879c31eda0bd975b875951654bf74d5231a34d00487b28714979e50fb" Nov 03 20:18:00.452204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.451834 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" event={"ID":"226bf833-ba6d-4c44-91bd-015817c60b68","Type":"ContainerStarted","Data":"c1fc201a572eb5617d3528d20b5643fb3e469ddeda8104b01205a9516000632b"} Nov 03 20:18:00.453791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.453765 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" event={"ID":"30c48c95-5c93-4887-86bb-8e2b76c120bf","Type":"ContainerStarted","Data":"281d4a37f07b025f0e770cdd92fedd7d98fe780fcfad967340d419454b9dcfa2"} Nov 03 20:18:00.455286 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.455260 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" event={"ID":"48a90be1-f894-4f95-8ccc-f8d2bd2d929d","Type":"ContainerDied","Data":"7ee22f14e131102f3774ea2271556cde9479a0f8617e3aacbbbb33ea8c51aaeb"} Nov 03 20:18:00.455286 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.455281 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk" Nov 03 20:18:00.464215 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.464089 5111 scope.go:122] "RemoveContainer" containerID="ff970f9879c31eda0bd975b875951654bf74d5231a34d00487b28714979e50fb" Nov 03 20:18:00.465303 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:18:00.465278 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff970f9879c31eda0bd975b875951654bf74d5231a34d00487b28714979e50fb\": container with ID starting with ff970f9879c31eda0bd975b875951654bf74d5231a34d00487b28714979e50fb not found: ID does not exist" containerID="ff970f9879c31eda0bd975b875951654bf74d5231a34d00487b28714979e50fb" Nov 03 20:18:00.465392 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.465314 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff970f9879c31eda0bd975b875951654bf74d5231a34d00487b28714979e50fb"} err="failed to get container status \"ff970f9879c31eda0bd975b875951654bf74d5231a34d00487b28714979e50fb\": rpc error: code = NotFound desc = could not find container \"ff970f9879c31eda0bd975b875951654bf74d5231a34d00487b28714979e50fb\": container with ID starting with ff970f9879c31eda0bd975b875951654bf74d5231a34d00487b28714979e50fb not found: ID does not exist" Nov 03 20:18:00.465392 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.465335 5111 scope.go:122] "RemoveContainer" containerID="3da3f0cbd0993adee23c0349751ff21b3ea92ed64537be82304bfeb73f7be019" Nov 03 20:18:00.476582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.472183 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv"] Nov 03 20:18:00.479947 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.479920 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-pwbwv"] Nov 03 20:18:00.493273 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.493247 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk"] Nov 03 20:18:00.500663 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:00.500633 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dnvxk"] Nov 03 20:18:01.484166 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:01.484124 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" event={"ID":"226bf833-ba6d-4c44-91bd-015817c60b68","Type":"ContainerStarted","Data":"f6248dfd3770b44252cc97107410e9e9182c70851aaaaba6871bb8810d841edb"} Nov 03 20:18:01.503397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:01.493333 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" event={"ID":"30c48c95-5c93-4887-86bb-8e2b76c120bf","Type":"ContainerStarted","Data":"958d342d9e3d0f5b764ac4389a7c94be62f916c4d6365d2c92ae7838f972db26"} Nov 03 20:18:01.514887 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:01.514823 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" podStartSLOduration=145.514802544 podStartE2EDuration="2m25.514802544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:01.507037138 +0000 UTC m=+2311.469415578" watchObservedRunningTime="2026-11-03 20:18:01.514802544 +0000 UTC m=+2311.477180983" Nov 03 20:18:01.528706 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:01.528628 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" podStartSLOduration=145.528610728 podStartE2EDuration="2m25.528610728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:01.52767223 +0000 UTC m=+2311.490050670" watchObservedRunningTime="2026-11-03 20:18:01.528610728 +0000 UTC m=+2311.490989167" Nov 03 20:18:02.010236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:02.010193 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:18:02.011850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:02.011820 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:18:02.011982 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:02.011874 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" podUID="30c48c95-5c93-4887-86bb-8e2b76c120bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:18:02.093869 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:02.093818 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:18:02.095579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:02.095526 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Nov 03 20:18:02.095837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:02.095795 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" podUID="226bf833-ba6d-4c44-91bd-015817c60b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Nov 03 20:18:02.228135 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:02.228090 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a90be1-f894-4f95-8ccc-f8d2bd2d929d" path="/var/lib/kubelet/pods/48a90be1-f894-4f95-8ccc-f8d2bd2d929d/volumes" Nov 03 20:18:02.232488 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:02.232445 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7310a5cf-4706-445f-8c50-95e2f57c1ee2" path="/var/lib/kubelet/pods/7310a5cf-4706-445f-8c50-95e2f57c1ee2/volumes" Nov 03 20:18:03.011018 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:03.010976 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:18:03.011608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:03.011056 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" podUID="30c48c95-5c93-4887-86bb-8e2b76c120bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:18:03.128500 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:03.128458 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Nov 03 20:18:03.129575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:03.128515 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" podUID="226bf833-ba6d-4c44-91bd-015817c60b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Nov 03 20:18:04.011019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:04.010979 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:18:04.011204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:04.011033 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" podUID="30c48c95-5c93-4887-86bb-8e2b76c120bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:18:04.104059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:04.103705 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Nov 03 20:18:04.104059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:04.103775 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" podUID="226bf833-ba6d-4c44-91bd-015817c60b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Nov 03 20:18:05.039649 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:05.024696 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:18:05.039649 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:05.024766 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" podUID="30c48c95-5c93-4887-86bb-8e2b76c120bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:18:05.093558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:05.091716 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Nov 03 20:18:05.093558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:05.091779 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" podUID="226bf833-ba6d-4c44-91bd-015817c60b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Nov 03 20:18:05.127661 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:18:05.127620 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:18:06.011504 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:06.010993 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:18:06.011504 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:06.011053 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" podUID="30c48c95-5c93-4887-86bb-8e2b76c120bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:18:06.094841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:06.094063 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Nov 03 20:18:06.094841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:06.094120 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" podUID="226bf833-ba6d-4c44-91bd-015817c60b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Nov 03 20:18:07.031077 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:07.030969 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:18:07.031077 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:07.031055 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" podUID="30c48c95-5c93-4887-86bb-8e2b76c120bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:18:07.111744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:07.105814 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Nov 03 20:18:07.111744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:07.105873 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" podUID="226bf833-ba6d-4c44-91bd-015817c60b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Nov 03 20:18:08.015281 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:08.015243 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:18:08.015771 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:08.015753 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:18:08.016444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:08.016423 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:18:08.096681 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:08.096646 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:18:08.097584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:08.097509 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:18:08.098464 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:08.098441 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:18:35.550907 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:18:35.550846 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:18:39.448459 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:39.448420 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65"] Nov 03 20:18:39.448984 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:39.448724 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" podUID="590bd9c8-3170-40be-89b4-15d8df5acd02" containerName="istio-proxy" containerID="cri-o://8356ad6cb6fa76a741dc59a31ea18a1f3f2c1041416517d93621d6dd8cf8a7fe" gracePeriod=30 Nov 03 20:18:44.875362 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.875334 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:18:44.911794 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.911754 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck"] Nov 03 20:18:44.913205 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.913181 5111 cpu_manager.go:414] "containerMap: removing container" podUID="590bd9c8-3170-40be-89b4-15d8df5acd02" containerName="istio-proxy" Nov 03 20:18:44.913283 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.913209 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="590bd9c8-3170-40be-89b4-15d8df5acd02" containerName="istio-proxy" Nov 03 20:18:44.913479 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.913463 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="590bd9c8-3170-40be-89b4-15d8df5acd02" containerName="istio-proxy" Nov 03 20:18:44.918429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.918404 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:44.929253 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.928274 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f817b0-openshift-default-dockercfg-wzqz5\"" Nov 03 20:18:44.934098 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.934068 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck"] Nov 03 20:18:44.972415 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.972377 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-workload-socket\") pod \"590bd9c8-3170-40be-89b4-15d8df5acd02\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " Nov 03 20:18:44.972683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.972476 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590bd9c8-3170-40be-89b4-15d8df5acd02-kube-api-access-2s868\" (UniqueName: \"kubernetes.io/projected/590bd9c8-3170-40be-89b4-15d8df5acd02-kube-api-access-2s868\") pod \"590bd9c8-3170-40be-89b4-15d8df5acd02\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " Nov 03 20:18:44.972683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.972591 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-envoy\") pod \"590bd9c8-3170-40be-89b4-15d8df5acd02\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " Nov 03 20:18:44.972683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.972627 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-podinfo\") pod \"590bd9c8-3170-40be-89b4-15d8df5acd02\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " Nov 03 20:18:44.972683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.972655 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-data\") pod \"590bd9c8-3170-40be-89b4-15d8df5acd02\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " Nov 03 20:18:44.973033 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.972838 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-credential-socket\") pod \"590bd9c8-3170-40be-89b4-15d8df5acd02\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " Nov 03 20:18:44.973033 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.972914 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-token\" (UniqueName: \"kubernetes.io/projected/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-token\") pod \"590bd9c8-3170-40be-89b4-15d8df5acd02\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " Nov 03 20:18:44.973033 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.972990 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/590bd9c8-3170-40be-89b4-15d8df5acd02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590bd9c8-3170-40be-89b4-15d8df5acd02-istiod-ca-cert\") pod \"590bd9c8-3170-40be-89b4-15d8df5acd02\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " Nov 03 20:18:44.974383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.973079 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-workload-certs\") pod \"590bd9c8-3170-40be-89b4-15d8df5acd02\" (UID: \"590bd9c8-3170-40be-89b4-15d8df5acd02\") " Nov 03 20:18:44.974383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.973382 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-data\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:44.974383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.973584 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-credential-socket\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:44.974383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.973616 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-credential-socket" pod "590bd9c8-3170-40be-89b4-15d8df5acd02" (UID: "590bd9c8-3170-40be-89b4-15d8df5acd02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:44.974383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.973647 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-token\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:44.974383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.973703 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istiod-ca-cert\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:44.974383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.973829 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-workload-certs" pod "590bd9c8-3170-40be-89b4-15d8df5acd02" (UID: "590bd9c8-3170-40be-89b4-15d8df5acd02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:44.974383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.973835 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-podinfo\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:44.974383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.973894 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-envoy\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:44.974383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.974008 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-workload-certs\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:44.974383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.974060 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qf6j\" (UniqueName: \"kubernetes.io/projected/a3e211eb-7ac1-4136-b8d0-65211bb869f6-kube-api-access-2qf6j\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:44.974383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.974118 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/590bd9c8-3170-40be-89b4-15d8df5acd02-istiod-ca-cert" pod "590bd9c8-3170-40be-89b4-15d8df5acd02" (UID: "590bd9c8-3170-40be-89b4-15d8df5acd02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:18:44.974383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.974338 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-workload-socket\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:44.976318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.974487 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:44.976318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.974505 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590bd9c8-3170-40be-89b4-15d8df5acd02-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:44.976318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.974521 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:44.976318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.974807 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-data" pod "590bd9c8-3170-40be-89b4-15d8df5acd02" (UID: "590bd9c8-3170-40be-89b4-15d8df5acd02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:44.976318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.975072 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-workload-socket" pod "590bd9c8-3170-40be-89b4-15d8df5acd02" (UID: "590bd9c8-3170-40be-89b4-15d8df5acd02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:44.977790 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.977766 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-podinfo" pod "590bd9c8-3170-40be-89b4-15d8df5acd02" (UID: "590bd9c8-3170-40be-89b4-15d8df5acd02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:18:44.977927 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.977837 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-token" pod "590bd9c8-3170-40be-89b4-15d8df5acd02" (UID: "590bd9c8-3170-40be-89b4-15d8df5acd02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:44.980075 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.980051 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-envoy" pod "590bd9c8-3170-40be-89b4-15d8df5acd02" (UID: "590bd9c8-3170-40be-89b4-15d8df5acd02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:44.981797 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:44.981772 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590bd9c8-3170-40be-89b4-15d8df5acd02-kube-api-access-2s868" pod "590bd9c8-3170-40be-89b4-15d8df5acd02" (UID: "590bd9c8-3170-40be-89b4-15d8df5acd02"). InnerVolumeSpecName "kube-api-access-2s868". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:45.076197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.076155 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-token\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.076416 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.076219 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istiod-ca-cert\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.076416 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.076304 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-podinfo\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.076416 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.076341 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-envoy\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.077684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.076419 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-workload-certs\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.077684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.076460 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qf6j\" (UniqueName: \"kubernetes.io/projected/a3e211eb-7ac1-4136-b8d0-65211bb869f6-kube-api-access-2qf6j\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.077684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.076591 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-workload-socket\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.077684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.076705 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-data\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.077684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.076859 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-credential-socket\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.077684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.076858 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-workload-certs\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.077684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.076976 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.077684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.076997 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s868\" (UniqueName: \"kubernetes.io/projected/590bd9c8-3170-40be-89b4-15d8df5acd02-kube-api-access-2s868\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.077684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.077021 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.077684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.077034 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.077684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.077039 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-workload-socket\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.077684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.077050 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-data\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.077684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.077047 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.077684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.077090 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590bd9c8-3170-40be-89b4-15d8df5acd02-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.077684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.077096 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istiod-ca-cert\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.077684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.077172 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-credential-socket\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.081242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.079002 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-envoy\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.081242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.079252 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-podinfo\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.086922 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.086698 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-token\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.087431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.087396 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qf6j\" (UniqueName: \"kubernetes.io/projected/a3e211eb-7ac1-4136-b8d0-65211bb869f6-kube-api-access-2qf6j\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.259976 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.259884 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:45.383349 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.383313 5111 generic.go:356] "Generic (PLEG): container finished" podID="590bd9c8-3170-40be-89b4-15d8df5acd02" containerID="8356ad6cb6fa76a741dc59a31ea18a1f3f2c1041416517d93621d6dd8cf8a7fe" exitCode=0 Nov 03 20:18:45.383527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.383363 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" Nov 03 20:18:45.383527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.383396 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" event={"ID":"590bd9c8-3170-40be-89b4-15d8df5acd02","Type":"ContainerDied","Data":"8356ad6cb6fa76a741dc59a31ea18a1f3f2c1041416517d93621d6dd8cf8a7fe"} Nov 03 20:18:45.383527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.383465 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" event={"ID":"590bd9c8-3170-40be-89b4-15d8df5acd02","Type":"ContainerDied","Data":"8a439770db976d0acb5db0abeab64ff6eb8704b54e468f70dd4dd435952fad53"} Nov 03 20:18:45.383527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.383492 5111 scope.go:122] "RemoveContainer" containerID="8356ad6cb6fa76a741dc59a31ea18a1f3f2c1041416517d93621d6dd8cf8a7fe" Nov 03 20:18:45.398652 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.398621 5111 scope.go:122] "RemoveContainer" containerID="8356ad6cb6fa76a741dc59a31ea18a1f3f2c1041416517d93621d6dd8cf8a7fe" Nov 03 20:18:45.402690 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:18:45.399086 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8356ad6cb6fa76a741dc59a31ea18a1f3f2c1041416517d93621d6dd8cf8a7fe\": container with ID starting with 8356ad6cb6fa76a741dc59a31ea18a1f3f2c1041416517d93621d6dd8cf8a7fe not found: ID does not exist" containerID="8356ad6cb6fa76a741dc59a31ea18a1f3f2c1041416517d93621d6dd8cf8a7fe" Nov 03 20:18:45.402690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.399126 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8356ad6cb6fa76a741dc59a31ea18a1f3f2c1041416517d93621d6dd8cf8a7fe"} err="failed to get container status \"8356ad6cb6fa76a741dc59a31ea18a1f3f2c1041416517d93621d6dd8cf8a7fe\": rpc error: code = NotFound desc = could not find container \"8356ad6cb6fa76a741dc59a31ea18a1f3f2c1041416517d93621d6dd8cf8a7fe\": container with ID starting with 8356ad6cb6fa76a741dc59a31ea18a1f3f2c1041416517d93621d6dd8cf8a7fe not found: ID does not exist" Nov 03 20:18:45.419226 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.419193 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65"] Nov 03 20:18:45.431428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.429600 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65"] Nov 03 20:18:45.600570 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:18:45.597352 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e211eb_7ac1_4136_b8d0_65211bb869f6.slice/crio-e3613d33dfca140eb0d426c1e636cc5e115db5f9b87377d09799db2996038e7d WatchSource:0}: Error finding container e3613d33dfca140eb0d426c1e636cc5e115db5f9b87377d09799db2996038e7d: Status 404 returned error can't find the container with id e3613d33dfca140eb0d426c1e636cc5e115db5f9b87377d09799db2996038e7d Nov 03 20:18:45.619633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.606666 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck"] Nov 03 20:18:45.635228 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.633271 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:18:45.635654 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.635613 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:18:45.635812 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.635717 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:18:45.821685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.821647 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:18:45.821845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:45.821708 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-dlf65" podUID="590bd9c8-3170-40be-89b4-15d8df5acd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:46.201867 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:46.201827 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="590bd9c8-3170-40be-89b4-15d8df5acd02" path="/var/lib/kubelet/pods/590bd9c8-3170-40be-89b4-15d8df5acd02/volumes" Nov 03 20:18:46.476705 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:46.476585 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" event={"ID":"a3e211eb-7ac1-4136-b8d0-65211bb869f6","Type":"ContainerStarted","Data":"846b3c01b9cb7f1dac233ae831f1943fad718283e588e39e894e632fec55f557"} Nov 03 20:18:46.476705 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:46.476651 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" event={"ID":"a3e211eb-7ac1-4136-b8d0-65211bb869f6","Type":"ContainerStarted","Data":"e3613d33dfca140eb0d426c1e636cc5e115db5f9b87377d09799db2996038e7d"} Nov 03 20:18:46.521523 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:46.521462 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" podStartSLOduration=29.52144617 podStartE2EDuration="29.52144617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:46.502333233 +0000 UTC m=+2356.464711696" watchObservedRunningTime="2026-11-03 20:18:46.52144617 +0000 UTC m=+2356.483824609" Nov 03 20:18:47.280799 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:47.280758 5111 patch_prober.go:29] interesting pod/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 20:18:47.281438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:47.281404 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" podUID="a3e211eb-7ac1-4136-b8d0-65211bb869f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 20:18:47.281836 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:47.281806 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:48.279190 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:48.278673 5111 patch_prober.go:29] interesting pod/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 20:18:48.279190 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:48.278744 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" podUID="a3e211eb-7ac1-4136-b8d0-65211bb869f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 20:18:49.268559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:49.265956 5111 patch_prober.go:29] interesting pod/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 20:18:49.268559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:49.266036 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" podUID="a3e211eb-7ac1-4136-b8d0-65211bb869f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 20:18:50.262862 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:50.262364 5111 patch_prober.go:29] interesting pod/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 20:18:50.262862 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:50.262439 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" podUID="a3e211eb-7ac1-4136-b8d0-65211bb869f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 20:18:51.264558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:51.263216 5111 patch_prober.go:29] interesting pod/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 20:18:51.264558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:51.263307 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" podUID="a3e211eb-7ac1-4136-b8d0-65211bb869f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 20:18:52.269944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:52.269901 5111 patch_prober.go:29] interesting pod/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 20:18:52.270436 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:52.269973 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" podUID="a3e211eb-7ac1-4136-b8d0-65211bb869f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 20:18:53.261478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:53.261440 5111 patch_prober.go:29] interesting pod/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 20:18:53.261695 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:53.261512 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" podUID="a3e211eb-7ac1-4136-b8d0-65211bb869f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 20:18:54.266966 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:54.266932 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:54.267490 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:54.267464 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:54.268207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:54.268186 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:18:56.963099 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:56.963058 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t"] Nov 03 20:18:56.982113 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:18:56.982078 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws"] Nov 03 20:19:01.964284 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:01.964223 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" podUID="ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7" containerName="payload-processing" containerID="cri-o://73baac09f3730e7252efed2bea1887ab5da8bb2ce6608f87997db9ddc894e8a3" gracePeriod=25 Nov 03 20:19:01.982783 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:01.982736 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" podUID="e6ef42c1-3bd0-4679-809b-72d16fe33a2f" containerName="payload-pre-processing" containerID="cri-o://f3a1d2c04b941c6ce0ba897066682c01990a1315ec615745b290b8b39c4dcfe4" gracePeriod=25 Nov 03 20:19:02.395518 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:02.395484 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" podUID="e6ef42c1-3bd0-4679-809b-72d16fe33a2f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:19:02.395727 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:02.395488 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" podUID="ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:19:05.061066 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:19:05.061032 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:19:06.734286 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:06.734246 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5"] Nov 03 20:19:06.735016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:06.734499 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" podUID="f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" containerName="istio-proxy" containerID="cri-o://6c56f4071ac822f088f3b10a8d07bb200c9b38b2914cbaf1d497f42b49b2078c" gracePeriod=30 Nov 03 20:19:06.820064 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:06.820028 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92"] Nov 03 20:19:06.838015 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:06.837983 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms"] Nov 03 20:19:11.820413 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:11.820377 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" podUID="e7b3d839-ce01-41bd-91fa-e1771134c8d8" containerName="payload-processing" containerID="cri-o://6284dc7fe7153a22090afeb26b33740bac59da7e861827bc2b03258abd30b797" gracePeriod=25 Nov 03 20:19:11.839254 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:11.839213 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" podUID="98bb8ea8-1345-4d95-80cf-43ac69ad4344" containerName="payload-pre-processing" containerID="cri-o://572f5404053d8969713a9bb8fdb264930ea235734fcdbd0cbf19a5238a91aa3a" gracePeriod=25 Nov 03 20:19:12.081107 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.081016 5111 generic.go:356] "Generic (PLEG): container finished" podID="f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" containerID="6c56f4071ac822f088f3b10a8d07bb200c9b38b2914cbaf1d497f42b49b2078c" exitCode=0 Nov 03 20:19:12.081285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.081206 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" event={"ID":"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9","Type":"ContainerDied","Data":"6c56f4071ac822f088f3b10a8d07bb200c9b38b2914cbaf1d497f42b49b2078c"} Nov 03 20:19:12.148509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.148472 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:19:12.185078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.180219 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-data\") pod \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " Nov 03 20:19:12.185078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.180284 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-kube-api-access-2phrl\" (UniqueName: \"kubernetes.io/projected/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-kube-api-access-2phrl\") pod \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " Nov 03 20:19:12.185078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.183324 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istiod-ca-cert\") pod \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " Nov 03 20:19:12.185078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.183422 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-workload-socket\") pod \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " Nov 03 20:19:12.185078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.183496 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-envoy\") pod \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " Nov 03 20:19:12.185078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.183584 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-workload-certs\") pod \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " Nov 03 20:19:12.185078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.183693 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-podinfo\") pod \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " Nov 03 20:19:12.185078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.183806 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-token\" (UniqueName: \"kubernetes.io/projected/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-token\") pod \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " Nov 03 20:19:12.185078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.184074 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-credential-socket\") pod \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\" (UID: \"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9\") " Nov 03 20:19:12.187217 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.187179 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-workload-socket" pod "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" (UID: "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:12.187644 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.187624 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:12.188179 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.188153 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-credential-socket" pod "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" (UID: "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:12.188717 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.188697 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-data" pod "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" (UID: "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:12.191017 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.190983 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-envoy" pod "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" (UID: "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:12.191230 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.191209 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-workload-certs" pod "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" (UID: "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:12.192867 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.192831 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istiod-ca-cert" pod "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" (UID: "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:12.202654 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.200760 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-kube-api-access-2phrl" pod "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" (UID: "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9"). InnerVolumeSpecName "kube-api-access-2phrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:12.211337 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.210738 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-podinfo" pod "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" (UID: "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:19:12.211337 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.211239 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-token" pod "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" (UID: "f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:12.246044 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.246022 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl"] Nov 03 20:19:12.247399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.247368 5111 cpu_manager.go:414] "containerMap: removing container" podUID="f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" containerName="istio-proxy" Nov 03 20:19:12.247399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.247392 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" containerName="istio-proxy" Nov 03 20:19:12.247800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.247777 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" containerName="istio-proxy" Nov 03 20:19:12.254596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.252343 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl"] Nov 03 20:19:12.254596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.252516 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.258779 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.258757 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-46c34c-openshift-default-dockercfg-55bnp\"" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.289820 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-workload-certs\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.290032 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbrbt\" (UniqueName: \"kubernetes.io/projected/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-kube-api-access-mbrbt\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.290106 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-podinfo\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.290279 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-workload-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.290355 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-envoy\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.291131 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-credential-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.291261 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-token\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.291298 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-data\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.291370 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istiod-ca-cert\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.291596 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.291614 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.291624 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.291633 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.291643 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.291652 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.291661 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2phrl\" (UniqueName: \"kubernetes.io/projected/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-kube-api-access-2phrl\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:12.295600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.291673 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:12.393634 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.393584 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-workload-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.393845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.393683 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-envoy\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.393922 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.393880 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-credential-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.394005 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.393982 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-token\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.394151 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.394013 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-workload-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.394151 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.394019 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-data\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.394151 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.394070 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istiod-ca-cert\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.394340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.394155 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-workload-certs\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.394340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.394226 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-credential-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.394340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.394229 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbrbt\" (UniqueName: \"kubernetes.io/projected/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-kube-api-access-mbrbt\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.394340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.394308 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-podinfo\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.394715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.394678 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-data\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.394789 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.394771 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-workload-certs\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.394909 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.394883 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istiod-ca-cert\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.396301 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.396268 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" podUID="ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:19:12.396699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.396664 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-envoy\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.397289 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.397264 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" podUID="e6ef42c1-3bd0-4679-809b-72d16fe33a2f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:19:12.397289 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.397279 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-podinfo\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.402274 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.402250 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-token\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.402382 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.402364 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbrbt\" (UniqueName: \"kubernetes.io/projected/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-kube-api-access-mbrbt\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.573812 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.573727 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:12.938911 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:19:12.938876 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de95fa5_fdc8_4b2b_bbe9_a40b2d4672c7.slice/crio-8b328fc7e8f43d8139ffe5ccbe737247dfa5e3e0c1db6283897e930d9755cc62 WatchSource:0}: Error finding container 8b328fc7e8f43d8139ffe5ccbe737247dfa5e3e0c1db6283897e930d9755cc62: Status 404 returned error can't find the container with id 8b328fc7e8f43d8139ffe5ccbe737247dfa5e3e0c1db6283897e930d9755cc62 Nov 03 20:19:12.939270 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.939035 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl"] Nov 03 20:19:12.942223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.942189 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:19:12.942329 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.942255 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:19:12.942329 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:12.942285 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:19:13.092927 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:13.092896 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" Nov 03 20:19:13.093095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:13.092892 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5" event={"ID":"f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9","Type":"ContainerDied","Data":"8a3caeb2d8c691a666b4129ea30ee64e0ad56af4b568a8b477dd93a802302c68"} Nov 03 20:19:13.093095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:13.093081 5111 scope.go:122] "RemoveContainer" containerID="6c56f4071ac822f088f3b10a8d07bb200c9b38b2914cbaf1d497f42b49b2078c" Nov 03 20:19:13.096108 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:13.096078 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" event={"ID":"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7","Type":"ContainerStarted","Data":"dd41eef0cff8d6939e3a40aa2c2a1611017397716eb156b3af0ff32a20b29fd8"} Nov 03 20:19:13.096237 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:13.096154 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" event={"ID":"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7","Type":"ContainerStarted","Data":"8b328fc7e8f43d8139ffe5ccbe737247dfa5e3e0c1db6283897e930d9755cc62"} Nov 03 20:19:13.121596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:13.121545 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" podStartSLOduration=9.121516385 podStartE2EDuration="9.121516385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:13.119488493 +0000 UTC m=+2383.081866934" watchObservedRunningTime="2026-11-03 20:19:13.121516385 +0000 UTC m=+2383.083894824" Nov 03 20:19:13.139277 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:13.139244 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5"] Nov 03 20:19:13.144131 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:13.144106 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-sxdb5"] Nov 03 20:19:13.574496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:13.574460 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:14.195590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:14.195499 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9" path="/var/lib/kubelet/pods/f854add6-2ad9-4b27-a0a2-72bf8f2d6fb9/volumes" Nov 03 20:19:14.575846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:14.575759 5111 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 20:19:14.576016 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:14.575850 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" podUID="8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 20:19:15.574437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:15.574398 5111 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 20:19:15.574911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:15.574459 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" podUID="8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 20:19:16.581047 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:16.581006 5111 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 20:19:16.599884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:16.599836 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" podUID="8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 20:19:16.978221 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:16.978188 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt"] Nov 03 20:19:16.978498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:16.978432 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" podUID="4de4cc37-85a3-41b1-b443-b52ccfce94d5" containerName="istio-proxy" containerID="cri-o://c71e341f6fc721a00f019a6deecd4c8c5ca238f61d919a369de26cbe7bd0b431" gracePeriod=30 Nov 03 20:19:17.579617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:17.579346 5111 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 20:19:17.579617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:17.579418 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" podUID="8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 20:19:17.638389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:17.633920 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw"] Nov 03 20:19:18.574858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:18.574823 5111 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 20:19:18.575148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:18.575109 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" podUID="8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 20:19:19.574812 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:19.574721 5111 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 20:19:19.574812 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:19.574775 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" podUID="8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 20:19:19.658115 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:19.658078 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" podUID="98bb8ea8-1345-4d95-80cf-43ac69ad4344" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:19:20.579303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:20.579272 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:20.579789 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:20.579679 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:20.580311 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:20.580280 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:19:20.723553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:20.723506 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" podUID="e7b3d839-ce01-41bd-91fa-e1771134c8d8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:19:22.389468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.389417 5111 generic.go:356] "Generic (PLEG): container finished" podID="ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7" containerID="73baac09f3730e7252efed2bea1887ab5da8bb2ce6608f87997db9ddc894e8a3" exitCode=0 Nov 03 20:19:22.390025 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.389491 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" event={"ID":"ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7","Type":"ContainerDied","Data":"73baac09f3730e7252efed2bea1887ab5da8bb2ce6608f87997db9ddc894e8a3"} Nov 03 20:19:22.391656 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.391633 5111 generic.go:356] "Generic (PLEG): container finished" podID="e6ef42c1-3bd0-4679-809b-72d16fe33a2f" containerID="f3a1d2c04b941c6ce0ba897066682c01990a1315ec615745b290b8b39c4dcfe4" exitCode=0 Nov 03 20:19:22.391834 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.391820 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" event={"ID":"e6ef42c1-3bd0-4679-809b-72d16fe33a2f","Type":"ContainerDied","Data":"f3a1d2c04b941c6ce0ba897066682c01990a1315ec615745b290b8b39c4dcfe4"} Nov 03 20:19:22.394585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.394524 5111 generic.go:356] "Generic (PLEG): container finished" podID="4de4cc37-85a3-41b1-b443-b52ccfce94d5" containerID="c71e341f6fc721a00f019a6deecd4c8c5ca238f61d919a369de26cbe7bd0b431" exitCode=0 Nov 03 20:19:22.394903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.394800 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" event={"ID":"4de4cc37-85a3-41b1-b443-b52ccfce94d5","Type":"ContainerDied","Data":"c71e341f6fc721a00f019a6deecd4c8c5ca238f61d919a369de26cbe7bd0b431"} Nov 03 20:19:22.442779 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.442741 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" Nov 03 20:19:22.532065 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.532034 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" Nov 03 20:19:22.537029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.537003 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:19:22.568602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.568469 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7-kube-api-access-7v8r2\" (UniqueName: \"kubernetes.io/projected/ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7-kube-api-access-7v8r2\") pod \"ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7\" (UID: \"ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7\") " Nov 03 20:19:22.568602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.568523 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7-plugins-config-volume\") pod \"ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7\" (UID: \"ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7\") " Nov 03 20:19:22.569010 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.568977 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7-plugins-config-volume" pod "ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7" (UID: "ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:22.569999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.569906 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:22.574999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.574973 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7-kube-api-access-7v8r2" pod "ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7" (UID: "ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7"). InnerVolumeSpecName "kube-api-access-7v8r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:22.596619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.596588 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn"] Nov 03 20:19:22.598683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.597780 5111 cpu_manager.go:414] "containerMap: removing container" podUID="4de4cc37-85a3-41b1-b443-b52ccfce94d5" containerName="istio-proxy" Nov 03 20:19:22.598683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.597803 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de4cc37-85a3-41b1-b443-b52ccfce94d5" containerName="istio-proxy" Nov 03 20:19:22.598683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.597832 5111 cpu_manager.go:414] "containerMap: removing container" podUID="ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7" containerName="payload-processing" Nov 03 20:19:22.598683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.597840 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7" containerName="payload-processing" Nov 03 20:19:22.598683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.597856 5111 cpu_manager.go:414] "containerMap: removing container" podUID="e6ef42c1-3bd0-4679-809b-72d16fe33a2f" containerName="payload-pre-processing" Nov 03 20:19:22.598683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.597864 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ef42c1-3bd0-4679-809b-72d16fe33a2f" containerName="payload-pre-processing" Nov 03 20:19:22.598683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.598127 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7" containerName="payload-processing" Nov 03 20:19:22.598683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.598150 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ef42c1-3bd0-4679-809b-72d16fe33a2f" containerName="payload-pre-processing" Nov 03 20:19:22.598683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.598186 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de4cc37-85a3-41b1-b443-b52ccfce94d5" containerName="istio-proxy" Nov 03 20:19:22.610942 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.610919 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn"] Nov 03 20:19:22.611113 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.611031 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.634863 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.634834 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" podUID="fae340c1-f2b0-4d4f-ac20-8c2886e894b9" containerName="payload-processing" containerID="cri-o://c73792e820641395a277024b14e4395fd4c5f5e12026ccd4c7d6ba2bda766b88" gracePeriod=25 Nov 03 20:19:22.670771 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.670740 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de4cc37-85a3-41b1-b443-b52ccfce94d5-kube-api-access-269w4\" (UniqueName: \"kubernetes.io/projected/4de4cc37-85a3-41b1-b443-b52ccfce94d5-kube-api-access-269w4\") pod \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " Nov 03 20:19:22.670929 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.670851 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-token\" (UniqueName: \"kubernetes.io/projected/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-token\") pod \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " Nov 03 20:19:22.671053 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.671034 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-credential-socket\") pod \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " Nov 03 20:19:22.671131 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.671114 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-workload-socket\") pod \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " Nov 03 20:19:22.671239 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.671222 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ef42c1-3bd0-4679-809b-72d16fe33a2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ef42c1-3bd0-4679-809b-72d16fe33a2f-plugins-config-volume\") pod \"e6ef42c1-3bd0-4679-809b-72d16fe33a2f\" (UID: \"e6ef42c1-3bd0-4679-809b-72d16fe33a2f\") " Nov 03 20:19:22.671304 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.671286 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ef42c1-3bd0-4679-809b-72d16fe33a2f-kube-api-access-pmwhc\" (UniqueName: \"kubernetes.io/projected/e6ef42c1-3bd0-4679-809b-72d16fe33a2f-kube-api-access-pmwhc\") pod \"e6ef42c1-3bd0-4679-809b-72d16fe33a2f\" (UID: \"e6ef42c1-3bd0-4679-809b-72d16fe33a2f\") " Nov 03 20:19:22.671441 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.671425 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istiod-ca-cert\") pod \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " Nov 03 20:19:22.671516 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.671458 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-data\") pod \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " Nov 03 20:19:22.671636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.671513 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-workload-certs\") pod \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " Nov 03 20:19:22.671636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.671581 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-podinfo\") pod \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " Nov 03 20:19:22.671763 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.671711 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-envoy\") pod \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\" (UID: \"4de4cc37-85a3-41b1-b443-b52ccfce94d5\") " Nov 03 20:19:22.673424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.671858 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-workload-socket" pod "4de4cc37-85a3-41b1-b443-b52ccfce94d5" (UID: "4de4cc37-85a3-41b1-b443-b52ccfce94d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:22.673424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.671915 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ef42c1-3bd0-4679-809b-72d16fe33a2f-plugins-config-volume" pod "e6ef42c1-3bd0-4679-809b-72d16fe33a2f" (UID: "e6ef42c1-3bd0-4679-809b-72d16fe33a2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:22.673424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.672150 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-workload-certs" pod "4de4cc37-85a3-41b1-b443-b52ccfce94d5" (UID: "4de4cc37-85a3-41b1-b443-b52ccfce94d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:22.673424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.672602 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istiod-ca-cert" pod "4de4cc37-85a3-41b1-b443-b52ccfce94d5" (UID: "4de4cc37-85a3-41b1-b443-b52ccfce94d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:22.673424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.672706 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:22.673424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.672730 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:22.673424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.672744 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v8r2\" (UniqueName: \"kubernetes.io/projected/ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7-kube-api-access-7v8r2\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:22.673424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.672760 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ef42c1-3bd0-4679-809b-72d16fe33a2f-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:22.673424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.672798 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-credential-socket" pod "4de4cc37-85a3-41b1-b443-b52ccfce94d5" (UID: "4de4cc37-85a3-41b1-b443-b52ccfce94d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:22.673424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.673395 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-data" pod "4de4cc37-85a3-41b1-b443-b52ccfce94d5" (UID: "4de4cc37-85a3-41b1-b443-b52ccfce94d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:22.674338 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.674314 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de4cc37-85a3-41b1-b443-b52ccfce94d5-kube-api-access-269w4" pod "4de4cc37-85a3-41b1-b443-b52ccfce94d5" (UID: "4de4cc37-85a3-41b1-b443-b52ccfce94d5"). InnerVolumeSpecName "kube-api-access-269w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:22.674608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.674498 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-token" pod "4de4cc37-85a3-41b1-b443-b52ccfce94d5" (UID: "4de4cc37-85a3-41b1-b443-b52ccfce94d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:22.674717 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.674673 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-envoy" pod "4de4cc37-85a3-41b1-b443-b52ccfce94d5" (UID: "4de4cc37-85a3-41b1-b443-b52ccfce94d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:22.675989 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.675962 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-podinfo" pod "4de4cc37-85a3-41b1-b443-b52ccfce94d5" (UID: "4de4cc37-85a3-41b1-b443-b52ccfce94d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:19:22.676205 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.676184 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ef42c1-3bd0-4679-809b-72d16fe33a2f-kube-api-access-pmwhc" pod "e6ef42c1-3bd0-4679-809b-72d16fe33a2f" (UID: "e6ef42c1-3bd0-4679-809b-72d16fe33a2f"). InnerVolumeSpecName "kube-api-access-pmwhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:22.774977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.774224 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-workload-certs\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.774977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.774346 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7fc530f-7898-450f-bc05-be85b74215cf-istio-token\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.774977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.774660 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-istio-envoy\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.774977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.774719 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7fc530f-7898-450f-bc05-be85b74215cf-istio-podinfo\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.774977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.774774 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-credential-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.774977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.774847 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-workload-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.775403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.775080 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-istio-data\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.775403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.775126 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7fc530f-7898-450f-bc05-be85b74215cf-istiod-ca-cert\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.775403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.775249 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwps5\" (UniqueName: \"kubernetes.io/projected/b7fc530f-7898-450f-bc05-be85b74215cf-kube-api-access-cwps5\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.775403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.775357 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmwhc\" (UniqueName: \"kubernetes.io/projected/e6ef42c1-3bd0-4679-809b-72d16fe33a2f-kube-api-access-pmwhc\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:22.775403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.775377 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:22.775403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.775392 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:22.775713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.775406 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:22.775713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.775422 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:22.775713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.775437 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-269w4\" (UniqueName: \"kubernetes.io/projected/4de4cc37-85a3-41b1-b443-b52ccfce94d5-kube-api-access-269w4\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:22.775713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.775451 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de4cc37-85a3-41b1-b443-b52ccfce94d5-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:22.775713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.775465 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4cc37-85a3-41b1-b443-b52ccfce94d5-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:22.876874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.876840 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-workload-certs\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.877699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.876958 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7fc530f-7898-450f-bc05-be85b74215cf-istio-token\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.877699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.877032 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-istio-envoy\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.877699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.877081 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7fc530f-7898-450f-bc05-be85b74215cf-istio-podinfo\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.877699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.877142 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-credential-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.877699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.877222 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-workload-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.877699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.877289 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-workload-certs\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.877699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.877373 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-istio-data\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.877699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.877418 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7fc530f-7898-450f-bc05-be85b74215cf-istiod-ca-cert\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.877699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.877564 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-workload-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.877699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.877622 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwps5\" (UniqueName: \"kubernetes.io/projected/b7fc530f-7898-450f-bc05-be85b74215cf-kube-api-access-cwps5\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.877699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.877664 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-credential-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.879427 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.877837 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-istio-data\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.879427 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.878745 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7fc530f-7898-450f-bc05-be85b74215cf-istiod-ca-cert\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.880482 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.880458 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-istio-envoy\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.881461 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.881443 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7fc530f-7898-450f-bc05-be85b74215cf-istio-podinfo\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.884980 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.884945 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7fc530f-7898-450f-bc05-be85b74215cf-istio-token\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.886201 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.886177 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwps5\" (UniqueName: \"kubernetes.io/projected/b7fc530f-7898-450f-bc05-be85b74215cf-kube-api-access-cwps5\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.910668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.910642 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" Nov 03 20:19:22.932033 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.932000 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:22.942539 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.942501 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl"] Nov 03 20:19:22.943495 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.943472 5111 cpu_manager.go:414] "containerMap: removing container" podUID="fae340c1-f2b0-4d4f-ac20-8c2886e894b9" containerName="payload-processing" Nov 03 20:19:22.943495 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.943489 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae340c1-f2b0-4d4f-ac20-8c2886e894b9" containerName="payload-processing" Nov 03 20:19:22.944190 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.943724 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae340c1-f2b0-4d4f-ac20-8c2886e894b9" containerName="payload-processing" Nov 03 20:19:22.953990 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.951672 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:22.958940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.958912 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl"] Nov 03 20:19:22.959840 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:22.959498 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-facc79-openshift-default-dockercfg-fjlx2\"" Nov 03 20:19:23.082004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.081963 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae340c1-f2b0-4d4f-ac20-8c2886e894b9-kube-api-access-snp4w\" (UniqueName: \"kubernetes.io/projected/fae340c1-f2b0-4d4f-ac20-8c2886e894b9-kube-api-access-snp4w\") pod \"fae340c1-f2b0-4d4f-ac20-8c2886e894b9\" (UID: \"fae340c1-f2b0-4d4f-ac20-8c2886e894b9\") " Nov 03 20:19:23.082224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.082052 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae340c1-f2b0-4d4f-ac20-8c2886e894b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae340c1-f2b0-4d4f-ac20-8c2886e894b9-plugins-config-volume\") pod \"fae340c1-f2b0-4d4f-ac20-8c2886e894b9\" (UID: \"fae340c1-f2b0-4d4f-ac20-8c2886e894b9\") " Nov 03 20:19:23.082421 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.082388 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae340c1-f2b0-4d4f-ac20-8c2886e894b9-plugins-config-volume" pod "fae340c1-f2b0-4d4f-ac20-8c2886e894b9" (UID: "fae340c1-f2b0-4d4f-ac20-8c2886e894b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:23.082817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.082795 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istiod-ca-cert\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.082930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.082860 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-podinfo\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.083040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.083025 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-credential-socket\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.083120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.083092 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-data\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.083182 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.083121 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-workload-certs\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.083242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.083190 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnhtl\" (UniqueName: \"kubernetes.io/projected/1a9914b3-5374-41b7-b530-480d2ec5a5ab-kube-api-access-fnhtl\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.083242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.083209 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-workload-socket\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.083357 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.083255 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-token\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.083357 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.083311 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-envoy\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.083490 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.083370 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae340c1-f2b0-4d4f-ac20-8c2886e894b9-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:23.086317 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.086278 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae340c1-f2b0-4d4f-ac20-8c2886e894b9-kube-api-access-snp4w" pod "fae340c1-f2b0-4d4f-ac20-8c2886e894b9" (UID: "fae340c1-f2b0-4d4f-ac20-8c2886e894b9"). InnerVolumeSpecName "kube-api-access-snp4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:23.193903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.192994 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-token\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.193903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.193115 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-envoy\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.193903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.193279 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istiod-ca-cert\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.193903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.193370 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-podinfo\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.193903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.193520 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-credential-socket\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.194684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.194656 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-data\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.196029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.194855 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-workload-certs\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.196029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.195098 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnhtl\" (UniqueName: \"kubernetes.io/projected/1a9914b3-5374-41b7-b530-480d2ec5a5ab-kube-api-access-fnhtl\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.196029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.195133 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-workload-socket\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.196029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.195255 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snp4w\" (UniqueName: \"kubernetes.io/projected/fae340c1-f2b0-4d4f-ac20-8c2886e894b9-kube-api-access-snp4w\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:23.196029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.195318 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-workload-certs\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.196029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.195634 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-workload-socket\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.197685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.196895 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-data\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.197830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.197794 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istiod-ca-cert\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.198326 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.198288 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-envoy\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.200744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.200487 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-podinfo\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.205071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.205040 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnhtl\" (UniqueName: \"kubernetes.io/projected/1a9914b3-5374-41b7-b530-480d2ec5a5ab-kube-api-access-fnhtl\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.209877 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.209854 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-credential-socket\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.213246 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.213209 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-token\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.264583 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:19:23.264088 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7fc530f_7898_450f_bc05_be85b74215cf.slice/crio-615746a7cd777027cc86233c54c87b5e4fbdc0d47615741f4b23f02da8b28920 WatchSource:0}: Error finding container 615746a7cd777027cc86233c54c87b5e4fbdc0d47615741f4b23f02da8b28920: Status 404 returned error can't find the container with id 615746a7cd777027cc86233c54c87b5e4fbdc0d47615741f4b23f02da8b28920 Nov 03 20:19:23.267355 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.267315 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:19:23.267480 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.267391 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:19:23.267480 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.267428 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:19:23.272025 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.271922 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn"] Nov 03 20:19:23.287711 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.287688 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:23.395173 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.394256 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" podUID="e6ef42c1-3bd0-4679-809b-72d16fe33a2f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.191:50052\" within 1s: context deadline exceeded" Nov 03 20:19:23.395173 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.394365 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" Nov 03 20:19:23.395173 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:19:23.394484 5111 logging.go:55] [core] [Channel #4952 SubChannel #4953]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.191:50052", ServerName: "10.132.0.191:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.191:50052: operation was canceled" Nov 03 20:19:23.395173 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.394827 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" podUID="ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.192:50052\" within 1s: context deadline exceeded" Nov 03 20:19:23.395173 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:19:23.394883 5111 logging.go:55] [core] [Channel #4950 SubChannel #4951]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.192:50052", ServerName: "10.132.0.192:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.192:50052: operation was canceled" Nov 03 20:19:23.395173 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.394909 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" Nov 03 20:19:23.407075 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.406560 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" Nov 03 20:19:23.407075 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.406592 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws" event={"ID":"e6ef42c1-3bd0-4679-809b-72d16fe33a2f","Type":"ContainerDied","Data":"1b7e6df9cad0865b3d0e26168fcea7d056631c96737c11c00d75879462edaf21"} Nov 03 20:19:23.407075 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.406666 5111 scope.go:122] "RemoveContainer" containerID="f3a1d2c04b941c6ce0ba897066682c01990a1315ec615745b290b8b39c4dcfe4" Nov 03 20:19:23.409339 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.408719 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" event={"ID":"b7fc530f-7898-450f-bc05-be85b74215cf","Type":"ContainerStarted","Data":"e075fee0e93ed3df9fc2733679b7f3e252cbb027768ec90c3872917b3a13ca05"} Nov 03 20:19:23.409339 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.408756 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" event={"ID":"b7fc530f-7898-450f-bc05-be85b74215cf","Type":"ContainerStarted","Data":"615746a7cd777027cc86233c54c87b5e4fbdc0d47615741f4b23f02da8b28920"} Nov 03 20:19:23.416469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.413253 5111 generic.go:356] "Generic (PLEG): container finished" podID="fae340c1-f2b0-4d4f-ac20-8c2886e894b9" containerID="c73792e820641395a277024b14e4395fd4c5f5e12026ccd4c7d6ba2bda766b88" exitCode=0 Nov 03 20:19:23.416469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.413371 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" event={"ID":"fae340c1-f2b0-4d4f-ac20-8c2886e894b9","Type":"ContainerDied","Data":"c73792e820641395a277024b14e4395fd4c5f5e12026ccd4c7d6ba2bda766b88"} Nov 03 20:19:23.416469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.413410 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" event={"ID":"fae340c1-f2b0-4d4f-ac20-8c2886e894b9","Type":"ContainerDied","Data":"1c46a350e8b9e15977bab4740a67c3f9d62e2dfebd3b4e48cb9b065aa9899b5a"} Nov 03 20:19:23.416469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.413474 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw" Nov 03 20:19:23.422947 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.422160 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" event={"ID":"4de4cc37-85a3-41b1-b443-b52ccfce94d5","Type":"ContainerDied","Data":"ddaa194a9ecf38883f4672b9480cb7192a7f702060cb7df8daa27606c9da3b25"} Nov 03 20:19:23.422947 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.422251 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt" Nov 03 20:19:23.425819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.425796 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" event={"ID":"ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7","Type":"ContainerDied","Data":"7f8780137ecafcfee4b9ea1fb5eaaddc8b2b81c3a3c41c0ddb3fd0fed8a49da8"} Nov 03 20:19:23.426036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.425983 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t" Nov 03 20:19:23.436982 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.433286 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" podStartSLOduration=19.43327076 podStartE2EDuration="19.43327076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:23.432015477 +0000 UTC m=+2393.394393928" watchObservedRunningTime="2026-11-03 20:19:23.43327076 +0000 UTC m=+2393.395649204" Nov 03 20:19:23.458068 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.458037 5111 scope.go:122] "RemoveContainer" containerID="c73792e820641395a277024b14e4395fd4c5f5e12026ccd4c7d6ba2bda766b88" Nov 03 20:19:23.480411 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.480027 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws"] Nov 03 20:19:23.480680 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.480659 5111 scope.go:122] "RemoveContainer" containerID="c73792e820641395a277024b14e4395fd4c5f5e12026ccd4c7d6ba2bda766b88" Nov 03 20:19:23.481418 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:19:23.481240 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c73792e820641395a277024b14e4395fd4c5f5e12026ccd4c7d6ba2bda766b88\": container with ID starting with c73792e820641395a277024b14e4395fd4c5f5e12026ccd4c7d6ba2bda766b88 not found: ID does not exist" containerID="c73792e820641395a277024b14e4395fd4c5f5e12026ccd4c7d6ba2bda766b88" Nov 03 20:19:23.481418 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.481284 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c73792e820641395a277024b14e4395fd4c5f5e12026ccd4c7d6ba2bda766b88"} err="failed to get container status \"c73792e820641395a277024b14e4395fd4c5f5e12026ccd4c7d6ba2bda766b88\": rpc error: code = NotFound desc = could not find container \"c73792e820641395a277024b14e4395fd4c5f5e12026ccd4c7d6ba2bda766b88\": container with ID starting with c73792e820641395a277024b14e4395fd4c5f5e12026ccd4c7d6ba2bda766b88 not found: ID does not exist" Nov 03 20:19:23.481418 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.481311 5111 scope.go:122] "RemoveContainer" containerID="c71e341f6fc721a00f019a6deecd4c8c5ca238f61d919a369de26cbe7bd0b431" Nov 03 20:19:23.491793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.491735 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49fb2b-86d58458bc-plfws"] Nov 03 20:19:23.499578 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.499514 5111 scope.go:122] "RemoveContainer" containerID="73baac09f3730e7252efed2bea1887ab5da8bb2ce6608f87997db9ddc894e8a3" Nov 03 20:19:23.502962 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.502938 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt"] Nov 03 20:19:23.512512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.512479 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-rdhkt"] Nov 03 20:19:23.544464 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.544427 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw"] Nov 03 20:19:23.554757 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.554724 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-facc79-6b465554d4-h46pw"] Nov 03 20:19:23.563227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.563201 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t"] Nov 03 20:19:23.574779 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.574744 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49fb2b-64ccbf5774-m6c8t"] Nov 03 20:19:23.591605 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:19:23.591571 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9914b3_5374_41b7_b530_480d2ec5a5ab.slice/crio-8293afa0875ef636278eec7cad8af6ecddc331c3daecf3589cdb22e226da757d WatchSource:0}: Error finding container 8293afa0875ef636278eec7cad8af6ecddc331c3daecf3589cdb22e226da757d: Status 404 returned error can't find the container with id 8293afa0875ef636278eec7cad8af6ecddc331c3daecf3589cdb22e226da757d Nov 03 20:19:23.593251 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.593222 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl"] Nov 03 20:19:23.595183 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.595144 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:19:23.595377 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.595356 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:19:23.595561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.595543 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:19:23.863374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.863277 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m"] Nov 03 20:19:23.863579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.863485 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" podUID="87cff690-9d53-44d6-a1dc-3f96238ca235" containerName="istio-proxy" containerID="cri-o://e62ecfa20290885ff5a59d4f58917bea7d380b6f21ce5da9bdf8187bb7dbf786" gracePeriod=30 Nov 03 20:19:23.932789 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:23.932755 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:24.134212 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:24.134171 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq"] Nov 03 20:19:24.134460 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:24.134429 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" podUID="75b8d7d9-7e56-4f0d-85f8-d5449b803007" containerName="istio-proxy" containerID="cri-o://0b982f7e87145ffd194b096b125bb399f8af4ffa676ab18a9f4ad56372b852c2" gracePeriod=30 Nov 03 20:19:24.203263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:24.203204 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de4cc37-85a3-41b1-b443-b52ccfce94d5" path="/var/lib/kubelet/pods/4de4cc37-85a3-41b1-b443-b52ccfce94d5/volumes" Nov 03 20:19:24.204239 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:24.204214 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7" path="/var/lib/kubelet/pods/ccef9d55-c21a-48c9-ad75-fc2e1d5e69c7/volumes" Nov 03 20:19:24.205183 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:24.205164 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ef42c1-3bd0-4679-809b-72d16fe33a2f" path="/var/lib/kubelet/pods/e6ef42c1-3bd0-4679-809b-72d16fe33a2f/volumes" Nov 03 20:19:24.206847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:24.206827 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae340c1-f2b0-4d4f-ac20-8c2886e894b9" path="/var/lib/kubelet/pods/fae340c1-f2b0-4d4f-ac20-8c2886e894b9/volumes" Nov 03 20:19:24.441803 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:24.441708 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" event={"ID":"1a9914b3-5374-41b7-b530-480d2ec5a5ab","Type":"ContainerStarted","Data":"94f4db1a819f549f5639d4a3d09326e27ee12cbcb2d3e2a905ccb0a70a067e99"} Nov 03 20:19:24.441803 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:24.441779 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" event={"ID":"1a9914b3-5374-41b7-b530-480d2ec5a5ab","Type":"ContainerStarted","Data":"8293afa0875ef636278eec7cad8af6ecddc331c3daecf3589cdb22e226da757d"} Nov 03 20:19:24.469694 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:24.469629 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" podStartSLOduration=64.469611152 podStartE2EDuration="1m4.469611152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:24.462129995 +0000 UTC m=+2394.424508434" watchObservedRunningTime="2026-11-03 20:19:24.469611152 +0000 UTC m=+2394.431989599" Nov 03 20:19:25.036570 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:24.986083 5111 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:19:25.155517 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:25.155365 5111 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:19:25.162830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:25.162785 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" podUID="75b8d7d9-7e56-4f0d-85f8-d5449b803007" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:19:25.163381 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:25.163347 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" podUID="b7fc530f-7898-450f-bc05-be85b74215cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:19:25.291089 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:25.290912 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:25.321266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:25.320792 5111 patch_prober.go:29] interesting pod/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 20:19:25.321266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:25.320872 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" podUID="1a9914b3-5374-41b7-b530-480d2ec5a5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 20:19:25.932813 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:25.932327 5111 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:19:25.932813 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:25.932388 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" podUID="b7fc530f-7898-450f-bc05-be85b74215cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:19:26.298613 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:26.293100 5111 patch_prober.go:29] interesting pod/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 20:19:26.298613 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:26.293169 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" podUID="1a9914b3-5374-41b7-b530-480d2ec5a5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 20:19:27.105611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:27.104342 5111 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:19:27.105611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:27.104403 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" podUID="b7fc530f-7898-450f-bc05-be85b74215cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:19:27.306224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:27.302198 5111 patch_prober.go:29] interesting pod/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 20:19:27.306224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:27.302259 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" podUID="1a9914b3-5374-41b7-b530-480d2ec5a5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 20:19:27.933186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:27.933150 5111 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:19:27.933363 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:27.933202 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" podUID="b7fc530f-7898-450f-bc05-be85b74215cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:19:28.292616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:28.290987 5111 patch_prober.go:29] interesting pod/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 20:19:28.292616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:28.291067 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" podUID="1a9914b3-5374-41b7-b530-480d2ec5a5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 20:19:28.977558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:28.970675 5111 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:19:28.977558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:28.970755 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" podUID="b7fc530f-7898-450f-bc05-be85b74215cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:19:29.289924 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.289887 5111 patch_prober.go:29] interesting pod/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 20:19:29.305146 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.304704 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" podUID="1a9914b3-5374-41b7-b530-480d2ec5a5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 20:19:29.442936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.439836 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:19:29.543049 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.542962 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-workload-certs\") pod \"87cff690-9d53-44d6-a1dc-3f96238ca235\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " Nov 03 20:19:29.543227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.543061 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-workload-socket\") pod \"87cff690-9d53-44d6-a1dc-3f96238ca235\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " Nov 03 20:19:29.543227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.543090 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-data\") pod \"87cff690-9d53-44d6-a1dc-3f96238ca235\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " Nov 03 20:19:29.543227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.543156 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-token\" (UniqueName: \"kubernetes.io/projected/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-token\") pod \"87cff690-9d53-44d6-a1dc-3f96238ca235\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " Nov 03 20:19:29.543227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.543212 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-envoy\") pod \"87cff690-9d53-44d6-a1dc-3f96238ca235\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " Nov 03 20:19:29.543449 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.543384 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-credential-socket\") pod \"87cff690-9d53-44d6-a1dc-3f96238ca235\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " Nov 03 20:19:29.543510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.543455 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87cff690-9d53-44d6-a1dc-3f96238ca235-kube-api-access-nzf9h\" (UniqueName: \"kubernetes.io/projected/87cff690-9d53-44d6-a1dc-3f96238ca235-kube-api-access-nzf9h\") pod \"87cff690-9d53-44d6-a1dc-3f96238ca235\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " Nov 03 20:19:29.543652 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.543628 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-podinfo\") pod \"87cff690-9d53-44d6-a1dc-3f96238ca235\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " Nov 03 20:19:29.543725 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.543710 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87cff690-9d53-44d6-a1dc-3f96238ca235-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cff690-9d53-44d6-a1dc-3f96238ca235-istiod-ca-cert\") pod \"87cff690-9d53-44d6-a1dc-3f96238ca235\" (UID: \"87cff690-9d53-44d6-a1dc-3f96238ca235\") " Nov 03 20:19:29.552811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.552747 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87cff690-9d53-44d6-a1dc-3f96238ca235-istiod-ca-cert" pod "87cff690-9d53-44d6-a1dc-3f96238ca235" (UID: "87cff690-9d53-44d6-a1dc-3f96238ca235"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:29.591558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.564630 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-workload-certs" pod "87cff690-9d53-44d6-a1dc-3f96238ca235" (UID: "87cff690-9d53-44d6-a1dc-3f96238ca235"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:29.591558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.572642 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-workload-socket" pod "87cff690-9d53-44d6-a1dc-3f96238ca235" (UID: "87cff690-9d53-44d6-a1dc-3f96238ca235"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:29.591558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.576611 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-data" pod "87cff690-9d53-44d6-a1dc-3f96238ca235" (UID: "87cff690-9d53-44d6-a1dc-3f96238ca235"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:29.601670 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.601633 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-credential-socket" pod "87cff690-9d53-44d6-a1dc-3f96238ca235" (UID: "87cff690-9d53-44d6-a1dc-3f96238ca235"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:29.631811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.614911 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87cff690-9d53-44d6-a1dc-3f96238ca235-kube-api-access-nzf9h" pod "87cff690-9d53-44d6-a1dc-3f96238ca235" (UID: "87cff690-9d53-44d6-a1dc-3f96238ca235"). InnerVolumeSpecName "kube-api-access-nzf9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:29.631811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.617924 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-token" pod "87cff690-9d53-44d6-a1dc-3f96238ca235" (UID: "87cff690-9d53-44d6-a1dc-3f96238ca235"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:29.631811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.618049 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-podinfo" pod "87cff690-9d53-44d6-a1dc-3f96238ca235" (UID: "87cff690-9d53-44d6-a1dc-3f96238ca235"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:19:29.631811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.618174 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-envoy" pod "87cff690-9d53-44d6-a1dc-3f96238ca235" (UID: "87cff690-9d53-44d6-a1dc-3f96238ca235"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:29.673940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.651571 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:29.673940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.651604 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cff690-9d53-44d6-a1dc-3f96238ca235-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:29.673940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.651618 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:29.673940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.651633 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:29.673940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.651652 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:29.822841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.651668 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:29.822841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.815618 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:29.822841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.815641 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cff690-9d53-44d6-a1dc-3f96238ca235-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:29.822841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.815659 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzf9h\" (UniqueName: \"kubernetes.io/projected/87cff690-9d53-44d6-a1dc-3f96238ca235-kube-api-access-nzf9h\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:29.925587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.923452 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" podUID="98bb8ea8-1345-4d95-80cf-43ac69ad4344" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:19:29.932310 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.932279 5111 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:19:29.932564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:29.932523 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" podUID="b7fc530f-7898-450f-bc05-be85b74215cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:19:30.026086 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.026039 5111 generic.go:356] "Generic (PLEG): container finished" podID="75b8d7d9-7e56-4f0d-85f8-d5449b803007" containerID="0b982f7e87145ffd194b096b125bb399f8af4ffa676ab18a9f4ad56372b852c2" exitCode=0 Nov 03 20:19:30.026262 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.026207 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" event={"ID":"75b8d7d9-7e56-4f0d-85f8-d5449b803007","Type":"ContainerDied","Data":"0b982f7e87145ffd194b096b125bb399f8af4ffa676ab18a9f4ad56372b852c2"} Nov 03 20:19:30.030711 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.030671 5111 generic.go:356] "Generic (PLEG): container finished" podID="87cff690-9d53-44d6-a1dc-3f96238ca235" containerID="e62ecfa20290885ff5a59d4f58917bea7d380b6f21ce5da9bdf8187bb7dbf786" exitCode=0 Nov 03 20:19:30.031275 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.031255 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" Nov 03 20:19:30.031744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.031720 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" event={"ID":"87cff690-9d53-44d6-a1dc-3f96238ca235","Type":"ContainerDied","Data":"e62ecfa20290885ff5a59d4f58917bea7d380b6f21ce5da9bdf8187bb7dbf786"} Nov 03 20:19:30.031868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.031784 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m" event={"ID":"87cff690-9d53-44d6-a1dc-3f96238ca235","Type":"ContainerDied","Data":"53f5703358a03e2fcc0d82de5fe63f2dee7ea86289a8e9f2b90e52419aa07aa9"} Nov 03 20:19:30.031868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.031815 5111 scope.go:122] "RemoveContainer" containerID="e62ecfa20290885ff5a59d4f58917bea7d380b6f21ce5da9bdf8187bb7dbf786" Nov 03 20:19:30.129881 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.129838 5111 scope.go:122] "RemoveContainer" containerID="e62ecfa20290885ff5a59d4f58917bea7d380b6f21ce5da9bdf8187bb7dbf786" Nov 03 20:19:30.132790 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:19:30.132760 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e62ecfa20290885ff5a59d4f58917bea7d380b6f21ce5da9bdf8187bb7dbf786\": container with ID starting with e62ecfa20290885ff5a59d4f58917bea7d380b6f21ce5da9bdf8187bb7dbf786 not found: ID does not exist" containerID="e62ecfa20290885ff5a59d4f58917bea7d380b6f21ce5da9bdf8187bb7dbf786" Nov 03 20:19:30.132891 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.132802 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e62ecfa20290885ff5a59d4f58917bea7d380b6f21ce5da9bdf8187bb7dbf786"} err="failed to get container status \"e62ecfa20290885ff5a59d4f58917bea7d380b6f21ce5da9bdf8187bb7dbf786\": rpc error: code = NotFound desc = could not find container \"e62ecfa20290885ff5a59d4f58917bea7d380b6f21ce5da9bdf8187bb7dbf786\": container with ID starting with e62ecfa20290885ff5a59d4f58917bea7d380b6f21ce5da9bdf8187bb7dbf786 not found: ID does not exist" Nov 03 20:19:30.291561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.266761 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:19:30.335824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.307572 5111 patch_prober.go:29] interesting pod/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 20:19:30.335824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.307807 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" podUID="1a9914b3-5374-41b7-b530-480d2ec5a5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 20:19:30.366557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.360196 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-podinfo\") pod \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " Nov 03 20:19:30.366557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.360288 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-credential-socket\") pod \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " Nov 03 20:19:30.366557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.360329 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istiod-ca-cert\") pod \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " Nov 03 20:19:30.366557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.360375 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-envoy\") pod \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " Nov 03 20:19:30.366557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.360490 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-token\" (UniqueName: \"kubernetes.io/projected/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-token\") pod \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " Nov 03 20:19:30.366557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.360589 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b8d7d9-7e56-4f0d-85f8-d5449b803007-kube-api-access-7667p\" (UniqueName: \"kubernetes.io/projected/75b8d7d9-7e56-4f0d-85f8-d5449b803007-kube-api-access-7667p\") pod \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " Nov 03 20:19:30.366557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.360775 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-workload-certs\") pod \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " Nov 03 20:19:30.366557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.360857 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-data\") pod \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " Nov 03 20:19:30.366557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.360890 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-workload-socket\") pod \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " Nov 03 20:19:30.395617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.379411 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-workload-certs" pod "75b8d7d9-7e56-4f0d-85f8-d5449b803007" (UID: "75b8d7d9-7e56-4f0d-85f8-d5449b803007"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:30.395617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.379758 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-credential-socket" pod "75b8d7d9-7e56-4f0d-85f8-d5449b803007" (UID: "75b8d7d9-7e56-4f0d-85f8-d5449b803007"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:30.395617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.380112 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istiod-ca-cert" pod "75b8d7d9-7e56-4f0d-85f8-d5449b803007" (UID: "75b8d7d9-7e56-4f0d-85f8-d5449b803007"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:30.395617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.383494 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-envoy" pod "75b8d7d9-7e56-4f0d-85f8-d5449b803007" (UID: "75b8d7d9-7e56-4f0d-85f8-d5449b803007"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:30.398126 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.398096 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m"] Nov 03 20:19:30.400642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.400617 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-hcd9m"] Nov 03 20:19:30.414023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.404743 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-token" pod "75b8d7d9-7e56-4f0d-85f8-d5449b803007" (UID: "75b8d7d9-7e56-4f0d-85f8-d5449b803007"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:30.414023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.405853 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-data" pod "75b8d7d9-7e56-4f0d-85f8-d5449b803007" (UID: "75b8d7d9-7e56-4f0d-85f8-d5449b803007"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:30.414341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.414302 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-workload-socket" pod "75b8d7d9-7e56-4f0d-85f8-d5449b803007" (UID: "75b8d7d9-7e56-4f0d-85f8-d5449b803007"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:30.467632 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.467599 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:30.472681 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.467864 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:30.472906 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.472885 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:30.473365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.473345 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:30.473505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.473490 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:30.473649 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.473633 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:30.473749 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.473737 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b8d7d9-7e56-4f0d-85f8-d5449b803007-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:30.535705 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.533893 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b8d7d9-7e56-4f0d-85f8-d5449b803007-kube-api-access-7667p" pod "75b8d7d9-7e56-4f0d-85f8-d5449b803007" (UID: "75b8d7d9-7e56-4f0d-85f8-d5449b803007"). InnerVolumeSpecName "kube-api-access-7667p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:30.617439 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.617372 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-podinfo" pod "75b8d7d9-7e56-4f0d-85f8-d5449b803007" (UID: "75b8d7d9-7e56-4f0d-85f8-d5449b803007"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:19:30.617643 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.617505 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-podinfo\") pod \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\" (UID: \"75b8d7d9-7e56-4f0d-85f8-d5449b803007\") " Nov 03 20:19:30.618973 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:19:30.618489 5111 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/75b8d7d9-7e56-4f0d-85f8-d5449b803007/volumes/kubernetes.io~downward-api/istio-podinfo Nov 03 20:19:30.618973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.618518 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-podinfo" pod "75b8d7d9-7e56-4f0d-85f8-d5449b803007" (UID: "75b8d7d9-7e56-4f0d-85f8-d5449b803007"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:19:30.618973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.618493 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7667p\" (UniqueName: \"kubernetes.io/projected/75b8d7d9-7e56-4f0d-85f8-d5449b803007-kube-api-access-7667p\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:30.618973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.618595 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b8d7d9-7e56-4f0d-85f8-d5449b803007-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:30.627099 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:19:30.627069 5111 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87cff690_9d53_44d6_a1dc_3f96238ca235.slice/crio-53f5703358a03e2fcc0d82de5fe63f2dee7ea86289a8e9f2b90e52419aa07aa9\": RecentStats: unable to find data in memory cache]" Nov 03 20:19:30.724902 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.724812 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" podUID="e7b3d839-ce01-41bd-91fa-e1771134c8d8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:19:30.824714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.824681 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:19:30.826063 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.826043 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:19:30.831586 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.831565 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:19:30.834043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.834021 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:19:30.932638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.932604 5111 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:19:30.932866 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:30.932845 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" podUID="b7fc530f-7898-450f-bc05-be85b74215cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:19:31.070685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:31.066864 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" event={"ID":"75b8d7d9-7e56-4f0d-85f8-d5449b803007","Type":"ContainerDied","Data":"53df79d223a01581d677d98c3736e432368cd69227f15db0cadd01edb4704ff5"} Nov 03 20:19:31.070685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:31.066949 5111 scope.go:122] "RemoveContainer" containerID="0b982f7e87145ffd194b096b125bb399f8af4ffa676ab18a9f4ad56372b852c2" Nov 03 20:19:31.070685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:31.067089 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq" Nov 03 20:19:31.114743 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:31.114702 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq"] Nov 03 20:19:31.123743 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:31.123713 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-sltgq"] Nov 03 20:19:31.289416 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:31.289151 5111 patch_prober.go:29] interesting pod/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 20:19:31.289416 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:31.289237 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" podUID="1a9914b3-5374-41b7-b530-480d2ec5a5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 20:19:31.941063 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:31.941034 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:31.941580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:31.941560 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:31.942497 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:31.942473 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:19:32.149590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.149503 5111 generic.go:356] "Generic (PLEG): container finished" podID="98bb8ea8-1345-4d95-80cf-43ac69ad4344" containerID="572f5404053d8969713a9bb8fdb264930ea235734fcdbd0cbf19a5238a91aa3a" exitCode=0 Nov 03 20:19:32.149867 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.149843 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" event={"ID":"98bb8ea8-1345-4d95-80cf-43ac69ad4344","Type":"ContainerDied","Data":"572f5404053d8969713a9bb8fdb264930ea235734fcdbd0cbf19a5238a91aa3a"} Nov 03 20:19:32.154280 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.153061 5111 generic.go:356] "Generic (PLEG): container finished" podID="e7b3d839-ce01-41bd-91fa-e1771134c8d8" containerID="6284dc7fe7153a22090afeb26b33740bac59da7e861827bc2b03258abd30b797" exitCode=0 Nov 03 20:19:32.154280 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.153271 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" event={"ID":"e7b3d839-ce01-41bd-91fa-e1771134c8d8","Type":"ContainerDied","Data":"6284dc7fe7153a22090afeb26b33740bac59da7e861827bc2b03258abd30b797"} Nov 03 20:19:32.203704 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.203667 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b8d7d9-7e56-4f0d-85f8-d5449b803007" path="/var/lib/kubelet/pods/75b8d7d9-7e56-4f0d-85f8-d5449b803007/volumes" Nov 03 20:19:32.204821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.204795 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87cff690-9d53-44d6-a1dc-3f96238ca235" path="/var/lib/kubelet/pods/87cff690-9d53-44d6-a1dc-3f96238ca235/volumes" Nov 03 20:19:32.385991 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.385923 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs"] Nov 03 20:19:32.387941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.387895 5111 cpu_manager.go:414] "containerMap: removing container" podUID="75b8d7d9-7e56-4f0d-85f8-d5449b803007" containerName="istio-proxy" Nov 03 20:19:32.387941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.387924 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b8d7d9-7e56-4f0d-85f8-d5449b803007" containerName="istio-proxy" Nov 03 20:19:32.388128 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.387946 5111 cpu_manager.go:414] "containerMap: removing container" podUID="87cff690-9d53-44d6-a1dc-3f96238ca235" containerName="istio-proxy" Nov 03 20:19:32.388128 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.387954 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87cff690-9d53-44d6-a1dc-3f96238ca235" containerName="istio-proxy" Nov 03 20:19:32.388293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.388274 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b8d7d9-7e56-4f0d-85f8-d5449b803007" containerName="istio-proxy" Nov 03 20:19:32.388359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.388303 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="87cff690-9d53-44d6-a1dc-3f96238ca235" containerName="istio-proxy" Nov 03 20:19:32.426563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.420830 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" Nov 03 20:19:32.663611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.644666 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7b3d839-ce01-41bd-91fa-e1771134c8d8-kube-api-access-c8glc\" (UniqueName: \"kubernetes.io/projected/e7b3d839-ce01-41bd-91fa-e1771134c8d8-kube-api-access-c8glc\") pod \"e7b3d839-ce01-41bd-91fa-e1771134c8d8\" (UID: \"e7b3d839-ce01-41bd-91fa-e1771134c8d8\") " Nov 03 20:19:32.663611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.644800 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7b3d839-ce01-41bd-91fa-e1771134c8d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7b3d839-ce01-41bd-91fa-e1771134c8d8-plugins-config-volume\") pod \"e7b3d839-ce01-41bd-91fa-e1771134c8d8\" (UID: \"e7b3d839-ce01-41bd-91fa-e1771134c8d8\") " Nov 03 20:19:32.812834 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.812790 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b3d839-ce01-41bd-91fa-e1771134c8d8-plugins-config-volume" pod "e7b3d839-ce01-41bd-91fa-e1771134c8d8" (UID: "e7b3d839-ce01-41bd-91fa-e1771134c8d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:32.813235 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.813209 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b3d839-ce01-41bd-91fa-e1771134c8d8-kube-api-access-c8glc" pod "e7b3d839-ce01-41bd-91fa-e1771134c8d8" (UID: "e7b3d839-ce01-41bd-91fa-e1771134c8d8"). InnerVolumeSpecName "kube-api-access-c8glc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:32.859759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.839846 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs"] Nov 03 20:19:32.859759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.840005 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx"] Nov 03 20:19:32.859759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.843711 5111 cpu_manager.go:414] "containerMap: removing container" podUID="e7b3d839-ce01-41bd-91fa-e1771134c8d8" containerName="payload-processing" Nov 03 20:19:32.859759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.843737 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7b3d839-ce01-41bd-91fa-e1771134c8d8" containerName="payload-processing" Nov 03 20:19:32.859759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.844052 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7b3d839-ce01-41bd-91fa-e1771134c8d8" containerName="payload-processing" Nov 03 20:19:32.859759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.854644 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" Nov 03 20:19:32.874574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.865629 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx"] Nov 03 20:19:32.874574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.865749 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" Nov 03 20:19:32.874574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.866017 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" Nov 03 20:19:32.874574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.870103 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-27d3ad-dockercfg-n9vhj\"" Nov 03 20:19:32.882557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.880561 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-27d3ad\"" Nov 03 20:19:32.989908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.981675 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7b3d839-ce01-41bd-91fa-e1771134c8d8-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:32.989908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:32.981702 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8glc\" (UniqueName: \"kubernetes.io/projected/e7b3d839-ce01-41bd-91fa-e1771134c8d8-kube-api-access-c8glc\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:33.035757 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.035704 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:33.037025 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.036982 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:33.058583 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.057793 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:33.082918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.082884 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98bb8ea8-1345-4d95-80cf-43ac69ad4344-kube-api-access-7hzm4\" (UniqueName: \"kubernetes.io/projected/98bb8ea8-1345-4d95-80cf-43ac69ad4344-kube-api-access-7hzm4\") pod \"98bb8ea8-1345-4d95-80cf-43ac69ad4344\" (UID: \"98bb8ea8-1345-4d95-80cf-43ac69ad4344\") " Nov 03 20:19:33.083119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.082956 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98bb8ea8-1345-4d95-80cf-43ac69ad4344-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98bb8ea8-1345-4d95-80cf-43ac69ad4344-plugins-config-volume\") pod \"98bb8ea8-1345-4d95-80cf-43ac69ad4344\" (UID: \"98bb8ea8-1345-4d95-80cf-43ac69ad4344\") " Nov 03 20:19:33.083323 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.083288 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92qvm\" (UniqueName: \"kubernetes.io/projected/b8748fcc-2ada-4d73-afd8-4ebd35ee60f8-kube-api-access-92qvm\") pod \"payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx\" (UID: \"b8748fcc-2ada-4d73-afd8-4ebd35ee60f8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" Nov 03 20:19:33.083434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.083324 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98bb8ea8-1345-4d95-80cf-43ac69ad4344-plugins-config-volume" pod "98bb8ea8-1345-4d95-80cf-43ac69ad4344" (UID: "98bb8ea8-1345-4d95-80cf-43ac69ad4344"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:33.083485 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.083450 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f79a1068-5b22-4a40-8c5a-1565c2593369-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs\" (UID: \"f79a1068-5b22-4a40-8c5a-1565c2593369\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" Nov 03 20:19:33.083768 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.083743 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwqq2\" (UniqueName: \"kubernetes.io/projected/f79a1068-5b22-4a40-8c5a-1565c2593369-kube-api-access-zwqq2\") pod \"payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs\" (UID: \"f79a1068-5b22-4a40-8c5a-1565c2593369\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" Nov 03 20:19:33.083881 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.083839 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8748fcc-2ada-4d73-afd8-4ebd35ee60f8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx\" (UID: \"b8748fcc-2ada-4d73-afd8-4ebd35ee60f8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" Nov 03 20:19:33.084099 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.084079 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98bb8ea8-1345-4d95-80cf-43ac69ad4344-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:33.086116 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.086080 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98bb8ea8-1345-4d95-80cf-43ac69ad4344-kube-api-access-7hzm4" pod "98bb8ea8-1345-4d95-80cf-43ac69ad4344" (UID: "98bb8ea8-1345-4d95-80cf-43ac69ad4344"). InnerVolumeSpecName "kube-api-access-7hzm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:33.168370 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.168341 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" Nov 03 20:19:33.168689 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.168377 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92" event={"ID":"e7b3d839-ce01-41bd-91fa-e1771134c8d8","Type":"ContainerDied","Data":"6181f3bbd50fdc40d08c8a4f41e67088c69b1f802a1d00b4db04e8763d54aaf2"} Nov 03 20:19:33.168689 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.168445 5111 scope.go:122] "RemoveContainer" containerID="6284dc7fe7153a22090afeb26b33740bac59da7e861827bc2b03258abd30b797" Nov 03 20:19:33.170340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.170307 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" Nov 03 20:19:33.170472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.170402 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms" event={"ID":"98bb8ea8-1345-4d95-80cf-43ac69ad4344","Type":"ContainerDied","Data":"64ef770bcc8268d3a4a87b3469e5bddd4125c1f2f18a5bff76f91d64dac0ddd9"} Nov 03 20:19:33.185362 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.185286 5111 scope.go:122] "RemoveContainer" containerID="572f5404053d8969713a9bb8fdb264930ea235734fcdbd0cbf19a5238a91aa3a" Nov 03 20:19:33.186177 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.186142 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92qvm\" (UniqueName: \"kubernetes.io/projected/b8748fcc-2ada-4d73-afd8-4ebd35ee60f8-kube-api-access-92qvm\") pod \"payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx\" (UID: \"b8748fcc-2ada-4d73-afd8-4ebd35ee60f8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" Nov 03 20:19:33.186483 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.186288 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f79a1068-5b22-4a40-8c5a-1565c2593369-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs\" (UID: \"f79a1068-5b22-4a40-8c5a-1565c2593369\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" Nov 03 20:19:33.186705 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.186595 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwqq2\" (UniqueName: \"kubernetes.io/projected/f79a1068-5b22-4a40-8c5a-1565c2593369-kube-api-access-zwqq2\") pod \"payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs\" (UID: \"f79a1068-5b22-4a40-8c5a-1565c2593369\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" Nov 03 20:19:33.186705 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.186689 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8748fcc-2ada-4d73-afd8-4ebd35ee60f8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx\" (UID: \"b8748fcc-2ada-4d73-afd8-4ebd35ee60f8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" Nov 03 20:19:33.186893 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.186832 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hzm4\" (UniqueName: \"kubernetes.io/projected/98bb8ea8-1345-4d95-80cf-43ac69ad4344-kube-api-access-7hzm4\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:33.188184 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.188160 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8748fcc-2ada-4d73-afd8-4ebd35ee60f8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx\" (UID: \"b8748fcc-2ada-4d73-afd8-4ebd35ee60f8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" Nov 03 20:19:33.188320 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.188223 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f79a1068-5b22-4a40-8c5a-1565c2593369-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs\" (UID: \"f79a1068-5b22-4a40-8c5a-1565c2593369\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" Nov 03 20:19:33.196282 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.196222 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92qvm\" (UniqueName: \"kubernetes.io/projected/b8748fcc-2ada-4d73-afd8-4ebd35ee60f8-kube-api-access-92qvm\") pod \"payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx\" (UID: \"b8748fcc-2ada-4d73-afd8-4ebd35ee60f8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" Nov 03 20:19:33.218362 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.207182 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwqq2\" (UniqueName: \"kubernetes.io/projected/f79a1068-5b22-4a40-8c5a-1565c2593369-kube-api-access-zwqq2\") pod \"payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs\" (UID: \"f79a1068-5b22-4a40-8c5a-1565c2593369\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" Nov 03 20:19:33.231915 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.231879 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92"] Nov 03 20:19:33.237400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.237366 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-43b780-b75f87544-ghs92"] Nov 03 20:19:33.242656 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.242625 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms"] Nov 03 20:19:33.247387 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.247300 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-43b780-cd6fc9f6c-ftdms"] Nov 03 20:19:33.293499 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.293472 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" Nov 03 20:19:33.309228 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.309201 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" Nov 03 20:19:33.645632 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.645599 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs"] Nov 03 20:19:33.647278 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:19:33.647238 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79a1068_5b22_4a40_8c5a_1565c2593369.slice/crio-597050db3a70b1acaa0adbbdd6e5ff01604b195608054756d077d3752011da39 WatchSource:0}: Error finding container 597050db3a70b1acaa0adbbdd6e5ff01604b195608054756d077d3752011da39: Status 404 returned error can't find the container with id 597050db3a70b1acaa0adbbdd6e5ff01604b195608054756d077d3752011da39 Nov 03 20:19:33.648074 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:19:33.648049 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8748fcc_2ada_4d73_afd8_4ebd35ee60f8.slice/crio-28dab73a53b02188e66d8d453603b0f6d7ccd66070e14347c8c029085860713a WatchSource:0}: Error finding container 28dab73a53b02188e66d8d453603b0f6d7ccd66070e14347c8c029085860713a: Status 404 returned error can't find the container with id 28dab73a53b02188e66d8d453603b0f6d7ccd66070e14347c8c029085860713a Nov 03 20:19:33.652791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:33.652766 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx"] Nov 03 20:19:34.198816 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:34.198770 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98bb8ea8-1345-4d95-80cf-43ac69ad4344" path="/var/lib/kubelet/pods/98bb8ea8-1345-4d95-80cf-43ac69ad4344/volumes" Nov 03 20:19:34.199427 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:34.199386 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7b3d839-ce01-41bd-91fa-e1771134c8d8" path="/var/lib/kubelet/pods/e7b3d839-ce01-41bd-91fa-e1771134c8d8/volumes" Nov 03 20:19:34.199821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:34.199800 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p"] Nov 03 20:19:34.199955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:34.199861 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" Nov 03 20:19:34.199955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:34.199882 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" event={"ID":"b8748fcc-2ada-4d73-afd8-4ebd35ee60f8","Type":"ContainerStarted","Data":"2294862985ddf6c8b3afeddbef29014a6ea9cc838b227f5ab40cfc8a32198411"} Nov 03 20:19:34.199955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:34.199902 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" Nov 03 20:19:34.199955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:34.199926 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" event={"ID":"b8748fcc-2ada-4d73-afd8-4ebd35ee60f8","Type":"ContainerStarted","Data":"28dab73a53b02188e66d8d453603b0f6d7ccd66070e14347c8c029085860713a"} Nov 03 20:19:34.200202 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:34.199982 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" event={"ID":"f79a1068-5b22-4a40-8c5a-1565c2593369","Type":"ContainerStarted","Data":"39351c34f51037b782b36380bd64c1490c1be7277f0b7a4c4a1642248c5c7e39"} Nov 03 20:19:34.200202 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:34.200027 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" event={"ID":"f79a1068-5b22-4a40-8c5a-1565c2593369","Type":"ContainerStarted","Data":"597050db3a70b1acaa0adbbdd6e5ff01604b195608054756d077d3752011da39"} Nov 03 20:19:34.200202 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:34.200060 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" podUID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerName="istio-proxy" containerID="cri-o://42ac199a7bb1fa7a351c74c38ae39ac877081c64bcebb1f417ab7b939396fd8a" gracePeriod=30 Nov 03 20:19:34.268501 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:34.268434 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" podStartSLOduration=2.26841621 podStartE2EDuration="2.26841621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:34.241927382 +0000 UTC m=+2404.204305853" watchObservedRunningTime="2026-11-03 20:19:34.26841621 +0000 UTC m=+2404.230794650" Nov 03 20:19:34.864783 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:34.864750 5111 scope.go:122] "RemoveContainer" containerID="886493d544a5573fea9f1739fc4cb83042e5b0bab7297edae130681a1f4e89d4" Nov 03 20:19:35.941406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:35.931766 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:36.005718 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.005649 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" podStartSLOduration=4.005628687 podStartE2EDuration="4.005628687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:34.27851327 +0000 UTC m=+2404.240891711" watchObservedRunningTime="2026-11-03 20:19:36.005628687 +0000 UTC m=+2405.968007127" Nov 03 20:19:36.070716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.027030 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:36.108453 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.038667 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 20:19:36.108821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.039787 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 03 20:19:36.123797 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.063611 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl"] Nov 03 20:19:36.126043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.126009 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" podUID="1a9914b3-5374-41b7-b530-480d2ec5a5ab" containerName="istio-proxy" containerID="cri-o://94f4db1a819f549f5639d4a3d09326e27ee12cbcb2d3e2a905ccb0a70a067e99" gracePeriod=30 Nov 03 20:19:36.126336 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.071199 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:36.323223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.288623 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:19:36.342004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.328735 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:36.441047 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:19:36.441017 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:19:36.526186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.526153 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn"] Nov 03 20:19:36.527795 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.527765 5111 cpu_manager.go:414] "containerMap: removing container" podUID="98bb8ea8-1345-4d95-80cf-43ac69ad4344" containerName="payload-pre-processing" Nov 03 20:19:36.527947 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.527934 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98bb8ea8-1345-4d95-80cf-43ac69ad4344" containerName="payload-pre-processing" Nov 03 20:19:36.528336 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.528318 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="98bb8ea8-1345-4d95-80cf-43ac69ad4344" containerName="payload-pre-processing" Nov 03 20:19:36.537345 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.537311 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn"] Nov 03 20:19:36.540320 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.539326 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" Nov 03 20:19:36.544342 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.544318 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg"] Nov 03 20:19:36.552201 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.552184 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg"] Nov 03 20:19:36.552289 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.552266 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" Nov 03 20:19:36.554002 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.553978 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv"] Nov 03 20:19:36.561036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.561012 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv"] Nov 03 20:19:36.561146 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.561126 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" Nov 03 20:19:36.571943 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.571916 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:19:36.572022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.572002 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:19:36.643134 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.643080 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4"] Nov 03 20:19:36.655327 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.654629 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvpb6\" (UniqueName: \"kubernetes.io/projected/25baca52-d88b-4838-be57-1975753451ed-kube-api-access-zvpb6\") pod \"maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg\" (UID: \"25baca52-d88b-4838-be57-1975753451ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" Nov 03 20:19:36.655723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.655691 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25baca52-d88b-4838-be57-1975753451ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg\" (UID: \"25baca52-d88b-4838-be57-1975753451ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" Nov 03 20:19:36.656191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.656149 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47274747-9jxqn\" (UID: \"386d1427-eb8e-4eba-b1ba-f0bac209dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" Nov 03 20:19:36.656795 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.656769 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25baca52-d88b-4838-be57-1975753451ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg\" (UID: \"25baca52-d88b-4838-be57-1975753451ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" Nov 03 20:19:36.661128 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.661073 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47274747-9jxqn\" (UID: \"386d1427-eb8e-4eba-b1ba-f0bac209dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" Nov 03 20:19:36.661303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.661283 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npcg8\" (UniqueName: \"kubernetes.io/projected/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-kube-api-access-npcg8\") pod \"maas-api-revoke-sub-47274747-9jxqn\" (UID: \"386d1427-eb8e-4eba-b1ba-f0bac209dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" Nov 03 20:19:36.674555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.671813 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4"] Nov 03 20:19:36.674555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.671906 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" Nov 03 20:19:36.763206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.763066 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25baca52-d88b-4838-be57-1975753451ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg\" (UID: \"25baca52-d88b-4838-be57-1975753451ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" Nov 03 20:19:36.763206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.763151 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e1b3b2-3628-4a69-a57a-60f399cc1038-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv\" (UID: \"98e1b3b2-3628-4a69-a57a-60f399cc1038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" Nov 03 20:19:36.763206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.763200 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47274747-9jxqn\" (UID: \"386d1427-eb8e-4eba-b1ba-f0bac209dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" Nov 03 20:19:36.764252 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.763234 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npcg8\" (UniqueName: \"kubernetes.io/projected/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-kube-api-access-npcg8\") pod \"maas-api-revoke-sub-47274747-9jxqn\" (UID: \"386d1427-eb8e-4eba-b1ba-f0bac209dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" Nov 03 20:19:36.764252 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.763386 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvpb6\" (UniqueName: \"kubernetes.io/projected/25baca52-d88b-4838-be57-1975753451ed-kube-api-access-zvpb6\") pod \"maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg\" (UID: \"25baca52-d88b-4838-be57-1975753451ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" Nov 03 20:19:36.764252 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.763478 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25baca52-d88b-4838-be57-1975753451ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg\" (UID: \"25baca52-d88b-4838-be57-1975753451ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" Nov 03 20:19:36.764252 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.763587 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e1b3b2-3628-4a69-a57a-60f399cc1038-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv\" (UID: \"98e1b3b2-3628-4a69-a57a-60f399cc1038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" Nov 03 20:19:36.764252 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.763693 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47274747-9jxqn\" (UID: \"386d1427-eb8e-4eba-b1ba-f0bac209dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" Nov 03 20:19:36.764252 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.763834 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnlp6\" (UniqueName: \"kubernetes.io/projected/98e1b3b2-3628-4a69-a57a-60f399cc1038-kube-api-access-wnlp6\") pod \"maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv\" (UID: \"98e1b3b2-3628-4a69-a57a-60f399cc1038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" Nov 03 20:19:36.764599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.764523 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25baca52-d88b-4838-be57-1975753451ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg\" (UID: \"25baca52-d88b-4838-be57-1975753451ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" Nov 03 20:19:36.765703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.765593 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47274747-9jxqn\" (UID: \"386d1427-eb8e-4eba-b1ba-f0bac209dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" Nov 03 20:19:36.768041 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.767967 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47274747-9jxqn\" (UID: \"386d1427-eb8e-4eba-b1ba-f0bac209dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" Nov 03 20:19:36.768143 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.768077 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25baca52-d88b-4838-be57-1975753451ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg\" (UID: \"25baca52-d88b-4838-be57-1975753451ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" Nov 03 20:19:36.773325 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.773301 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npcg8\" (UniqueName: \"kubernetes.io/projected/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-kube-api-access-npcg8\") pod \"maas-api-revoke-sub-47274747-9jxqn\" (UID: \"386d1427-eb8e-4eba-b1ba-f0bac209dfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" Nov 03 20:19:36.773759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.773735 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvpb6\" (UniqueName: \"kubernetes.io/projected/25baca52-d88b-4838-be57-1975753451ed-kube-api-access-zvpb6\") pod \"maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg\" (UID: \"25baca52-d88b-4838-be57-1975753451ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" Nov 03 20:19:36.865922 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.865872 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnps7\" (UniqueName: \"kubernetes.io/projected/21aa1610-bd37-4be6-8ef5-4217c931f3a5-kube-api-access-qnps7\") pod \"maas-api-revoke-sub-76598cd6-mt9j4\" (UID: \"21aa1610-bd37-4be6-8ef5-4217c931f3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" Nov 03 20:19:36.866141 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.866114 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e1b3b2-3628-4a69-a57a-60f399cc1038-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv\" (UID: \"98e1b3b2-3628-4a69-a57a-60f399cc1038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" Nov 03 20:19:36.866239 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.866227 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21aa1610-bd37-4be6-8ef5-4217c931f3a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76598cd6-mt9j4\" (UID: \"21aa1610-bd37-4be6-8ef5-4217c931f3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" Nov 03 20:19:36.866301 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.866285 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21aa1610-bd37-4be6-8ef5-4217c931f3a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76598cd6-mt9j4\" (UID: \"21aa1610-bd37-4be6-8ef5-4217c931f3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" Nov 03 20:19:36.866437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.866413 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnlp6\" (UniqueName: \"kubernetes.io/projected/98e1b3b2-3628-4a69-a57a-60f399cc1038-kube-api-access-wnlp6\") pod \"maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv\" (UID: \"98e1b3b2-3628-4a69-a57a-60f399cc1038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" Nov 03 20:19:36.866500 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.866451 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" Nov 03 20:19:36.866594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.866523 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e1b3b2-3628-4a69-a57a-60f399cc1038-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv\" (UID: \"98e1b3b2-3628-4a69-a57a-60f399cc1038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" Nov 03 20:19:36.866911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.866891 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e1b3b2-3628-4a69-a57a-60f399cc1038-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv\" (UID: \"98e1b3b2-3628-4a69-a57a-60f399cc1038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" Nov 03 20:19:36.869132 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.869110 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e1b3b2-3628-4a69-a57a-60f399cc1038-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv\" (UID: \"98e1b3b2-3628-4a69-a57a-60f399cc1038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" Nov 03 20:19:36.882975 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.882933 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnlp6\" (UniqueName: \"kubernetes.io/projected/98e1b3b2-3628-4a69-a57a-60f399cc1038-kube-api-access-wnlp6\") pod \"maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv\" (UID: \"98e1b3b2-3628-4a69-a57a-60f399cc1038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" Nov 03 20:19:36.894712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.894684 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" Nov 03 20:19:36.899407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.899095 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" Nov 03 20:19:36.968428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.968389 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21aa1610-bd37-4be6-8ef5-4217c931f3a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76598cd6-mt9j4\" (UID: \"21aa1610-bd37-4be6-8ef5-4217c931f3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" Nov 03 20:19:36.968911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.968492 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21aa1610-bd37-4be6-8ef5-4217c931f3a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76598cd6-mt9j4\" (UID: \"21aa1610-bd37-4be6-8ef5-4217c931f3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" Nov 03 20:19:36.968911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.968761 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnps7\" (UniqueName: \"kubernetes.io/projected/21aa1610-bd37-4be6-8ef5-4217c931f3a5-kube-api-access-qnps7\") pod \"maas-api-revoke-sub-76598cd6-mt9j4\" (UID: \"21aa1610-bd37-4be6-8ef5-4217c931f3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" Nov 03 20:19:36.969932 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.969906 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21aa1610-bd37-4be6-8ef5-4217c931f3a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76598cd6-mt9j4\" (UID: \"21aa1610-bd37-4be6-8ef5-4217c931f3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" Nov 03 20:19:36.992303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.989039 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21aa1610-bd37-4be6-8ef5-4217c931f3a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76598cd6-mt9j4\" (UID: \"21aa1610-bd37-4be6-8ef5-4217c931f3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" Nov 03 20:19:36.992303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:36.989884 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnps7\" (UniqueName: \"kubernetes.io/projected/21aa1610-bd37-4be6-8ef5-4217c931f3a5-kube-api-access-qnps7\") pod \"maas-api-revoke-sub-76598cd6-mt9j4\" (UID: \"21aa1610-bd37-4be6-8ef5-4217c931f3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" Nov 03 20:19:37.007767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:37.007463 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" Nov 03 20:19:37.195973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:37.195934 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn"] Nov 03 20:19:37.227915 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:37.227800 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg"] Nov 03 20:19:37.238740 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:37.238716 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv"] Nov 03 20:19:37.241992 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:19:37.241971 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e1b3b2_3628_4a69_a57a_60f399cc1038.slice/crio-a65195586b0676b4eeda8c429e3aa2bde46a6c4f19b39a2c970964a4c9181718 WatchSource:0}: Error finding container a65195586b0676b4eeda8c429e3aa2bde46a6c4f19b39a2c970964a4c9181718: Status 404 returned error can't find the container with id a65195586b0676b4eeda8c429e3aa2bde46a6c4f19b39a2c970964a4c9181718 Nov 03 20:19:37.314727 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:37.314691 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4"] Nov 03 20:19:37.350658 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:19:37.350626 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21aa1610_bd37_4be6_8ef5_4217c931f3a5.slice/crio-214ebf6473f1611d43fae66363eab1326c18641e01f8d9ec5de02dd98bb6a660 WatchSource:0}: Error finding container 214ebf6473f1611d43fae66363eab1326c18641e01f8d9ec5de02dd98bb6a660: Status 404 returned error can't find the container with id 214ebf6473f1611d43fae66363eab1326c18641e01f8d9ec5de02dd98bb6a660 Nov 03 20:19:38.009767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.009725 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:19:38.010274 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.009788 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" podUID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:19:38.124023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.123977 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" event={"ID":"21aa1610-bd37-4be6-8ef5-4217c931f3a5","Type":"ContainerStarted","Data":"2910d77601ca1a49b55f41e2377dbe1f821bd797a64ada2d3f8584495dd02752"} Nov 03 20:19:38.124228 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.124044 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" event={"ID":"21aa1610-bd37-4be6-8ef5-4217c931f3a5","Type":"ContainerStarted","Data":"214ebf6473f1611d43fae66363eab1326c18641e01f8d9ec5de02dd98bb6a660"} Nov 03 20:19:38.125862 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.125821 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" event={"ID":"25baca52-d88b-4838-be57-1975753451ed","Type":"ContainerStarted","Data":"1f17e5e23d97be66acb5b2f2819929f33fbd2cc4c1332bd8a6ec5d06deb84886"} Nov 03 20:19:38.126048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.125937 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" event={"ID":"25baca52-d88b-4838-be57-1975753451ed","Type":"ContainerStarted","Data":"d23fd49eb942add22887b2675ba2faa41559aac56321491dd2d6e3dd5a280e9a"} Nov 03 20:19:38.127621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.127592 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" event={"ID":"98e1b3b2-3628-4a69-a57a-60f399cc1038","Type":"ContainerStarted","Data":"e48a351c8b04aa9b84f50c99197d767c5663c641e0b0a4cbab4a1bf380f92d5b"} Nov 03 20:19:38.127770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.127646 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" event={"ID":"98e1b3b2-3628-4a69-a57a-60f399cc1038","Type":"ContainerStarted","Data":"a65195586b0676b4eeda8c429e3aa2bde46a6c4f19b39a2c970964a4c9181718"} Nov 03 20:19:38.129476 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.129452 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" event={"ID":"386d1427-eb8e-4eba-b1ba-f0bac209dfa9","Type":"ContainerStarted","Data":"bfa979a1e7b060995ab0aab437fd27b8feb3355901582b3d6d7ad3a6e4636f15"} Nov 03 20:19:38.129596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.129492 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" event={"ID":"386d1427-eb8e-4eba-b1ba-f0bac209dfa9","Type":"ContainerStarted","Data":"296f86736e0f7ccf118c1b81eb1aa0bbc6b577aaf4dab22ed7221ab3d6f8061f"} Nov 03 20:19:38.144515 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.144470 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" podStartSLOduration=2.144457091 podStartE2EDuration="2.144457091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:38.140853836 +0000 UTC m=+2408.103232267" watchObservedRunningTime="2026-11-03 20:19:38.144457091 +0000 UTC m=+2408.106835530" Nov 03 20:19:38.159426 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.159360 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" podStartSLOduration=2.1593474280000002 podStartE2EDuration="2.159347428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:38.156193097 +0000 UTC m=+2408.118571549" watchObservedRunningTime="2026-11-03 20:19:38.159347428 +0000 UTC m=+2408.121725866" Nov 03 20:19:38.174825 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.174741 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" podStartSLOduration=2.174725031 podStartE2EDuration="2.174725031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:38.171979786 +0000 UTC m=+2408.134358230" watchObservedRunningTime="2026-11-03 20:19:38.174725031 +0000 UTC m=+2408.137103472" Nov 03 20:19:38.191217 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.191166 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" podStartSLOduration=2.191153186 podStartE2EDuration="2.191153186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:38.189302862 +0000 UTC m=+2408.151681301" watchObservedRunningTime="2026-11-03 20:19:38.191153186 +0000 UTC m=+2408.153531670" Nov 03 20:19:38.718619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.717629 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7"] Nov 03 20:19:38.718619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.717894 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" podUID="a0da8389-92b0-42da-96cd-4dec2757e02a" containerName="istio-proxy" containerID="cri-o://243bd3b731e109ef7d8ee2279d84bb5d97405998fee26922447feeed7744b67b" gracePeriod=30 Nov 03 20:19:38.726374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.723331 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9"] Nov 03 20:19:38.726374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:38.723564 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" podUID="8f26fe8b-ab5c-4978-ab59-d163324180ff" containerName="istio-proxy" containerID="cri-o://8371bf3a77d805118c08fb37a0d865b00b7e22109a89f5b3d2821cb539b01fb4" gracePeriod=30 Nov 03 20:19:39.171197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.171156 5111 generic.go:356] "Generic (PLEG): container finished" podID="21aa1610-bd37-4be6-8ef5-4217c931f3a5" containerID="2910d77601ca1a49b55f41e2377dbe1f821bd797a64ada2d3f8584495dd02752" exitCode=0 Nov 03 20:19:39.171724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.171292 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" event={"ID":"21aa1610-bd37-4be6-8ef5-4217c931f3a5","Type":"ContainerDied","Data":"2910d77601ca1a49b55f41e2377dbe1f821bd797a64ada2d3f8584495dd02752"} Nov 03 20:19:39.174978 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.174947 5111 generic.go:356] "Generic (PLEG): container finished" podID="25baca52-d88b-4838-be57-1975753451ed" containerID="1f17e5e23d97be66acb5b2f2819929f33fbd2cc4c1332bd8a6ec5d06deb84886" exitCode=0 Nov 03 20:19:39.175135 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.175119 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" event={"ID":"25baca52-d88b-4838-be57-1975753451ed","Type":"ContainerDied","Data":"1f17e5e23d97be66acb5b2f2819929f33fbd2cc4c1332bd8a6ec5d06deb84886"} Nov 03 20:19:39.177199 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.177169 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt"] Nov 03 20:19:39.177831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.177785 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" podUID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerName="istio-proxy" containerID="cri-o://d093afa4d10294a4d4661b5d96263fc434fd51f7efeb98e86cc285976a039859" gracePeriod=30 Nov 03 20:19:39.178156 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.178127 5111 generic.go:356] "Generic (PLEG): container finished" podID="98e1b3b2-3628-4a69-a57a-60f399cc1038" containerID="e48a351c8b04aa9b84f50c99197d767c5663c641e0b0a4cbab4a1bf380f92d5b" exitCode=0 Nov 03 20:19:39.178626 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.178283 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" event={"ID":"98e1b3b2-3628-4a69-a57a-60f399cc1038","Type":"ContainerDied","Data":"e48a351c8b04aa9b84f50c99197d767c5663c641e0b0a4cbab4a1bf380f92d5b"} Nov 03 20:19:39.180633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.180608 5111 generic.go:356] "Generic (PLEG): container finished" podID="386d1427-eb8e-4eba-b1ba-f0bac209dfa9" containerID="bfa979a1e7b060995ab0aab437fd27b8feb3355901582b3d6d7ad3a6e4636f15" exitCode=0 Nov 03 20:19:39.180895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.180874 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" event={"ID":"386d1427-eb8e-4eba-b1ba-f0bac209dfa9","Type":"ContainerDied","Data":"bfa979a1e7b060995ab0aab437fd27b8feb3355901582b3d6d7ad3a6e4636f15"} Nov 03 20:19:39.553287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.553250 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l"] Nov 03 20:19:39.553505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.553476 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" podUID="8f272466-fdc2-4521-9f68-782e0cb8da27" containerName="istio-proxy" containerID="cri-o://3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39" gracePeriod=30 Nov 03 20:19:39.561263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.561236 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:19:39.648280 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.648248 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-workload-socket\") pod \"7de3942d-307c-4f4c-b291-2b69ad18f472\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " Nov 03 20:19:39.648443 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.648306 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de3942d-307c-4f4c-b291-2b69ad18f472-kube-api-access-t6djp\" (UniqueName: \"kubernetes.io/projected/7de3942d-307c-4f4c-b291-2b69ad18f472-kube-api-access-t6djp\") pod \"7de3942d-307c-4f4c-b291-2b69ad18f472\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " Nov 03 20:19:39.648443 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.648333 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-podinfo\") pod \"7de3942d-307c-4f4c-b291-2b69ad18f472\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " Nov 03 20:19:39.648603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.648471 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-workload-certs\") pod \"7de3942d-307c-4f4c-b291-2b69ad18f472\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " Nov 03 20:19:39.648661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.648646 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-credential-socket\") pod \"7de3942d-307c-4f4c-b291-2b69ad18f472\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " Nov 03 20:19:39.648810 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.648789 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-token\" (UniqueName: \"kubernetes.io/projected/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-token\") pod \"7de3942d-307c-4f4c-b291-2b69ad18f472\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " Nov 03 20:19:39.648912 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.648872 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7de3942d-307c-4f4c-b291-2b69ad18f472-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de3942d-307c-4f4c-b291-2b69ad18f472-istiod-ca-cert\") pod \"7de3942d-307c-4f4c-b291-2b69ad18f472\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " Nov 03 20:19:39.648912 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.648649 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-workload-socket" pod "7de3942d-307c-4f4c-b291-2b69ad18f472" (UID: "7de3942d-307c-4f4c-b291-2b69ad18f472"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:39.648912 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.648878 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-workload-certs" pod "7de3942d-307c-4f4c-b291-2b69ad18f472" (UID: "7de3942d-307c-4f4c-b291-2b69ad18f472"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:39.649097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.648997 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-credential-socket" pod "7de3942d-307c-4f4c-b291-2b69ad18f472" (UID: "7de3942d-307c-4f4c-b291-2b69ad18f472"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:39.649097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.649007 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-envoy\") pod \"7de3942d-307c-4f4c-b291-2b69ad18f472\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " Nov 03 20:19:39.649238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.649217 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-data\") pod \"7de3942d-307c-4f4c-b291-2b69ad18f472\" (UID: \"7de3942d-307c-4f4c-b291-2b69ad18f472\") " Nov 03 20:19:39.649367 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.649338 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de3942d-307c-4f4c-b291-2b69ad18f472-istiod-ca-cert" pod "7de3942d-307c-4f4c-b291-2b69ad18f472" (UID: "7de3942d-307c-4f4c-b291-2b69ad18f472"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:39.649601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.649574 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-data" pod "7de3942d-307c-4f4c-b291-2b69ad18f472" (UID: "7de3942d-307c-4f4c-b291-2b69ad18f472"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:39.650345 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.650315 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:39.650526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.650350 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de3942d-307c-4f4c-b291-2b69ad18f472-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:39.650526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.650365 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:39.650526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.650379 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:39.650526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.650392 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:39.651846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.651741 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-token" pod "7de3942d-307c-4f4c-b291-2b69ad18f472" (UID: "7de3942d-307c-4f4c-b291-2b69ad18f472"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:39.651846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.651795 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-podinfo" pod "7de3942d-307c-4f4c-b291-2b69ad18f472" (UID: "7de3942d-307c-4f4c-b291-2b69ad18f472"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:19:39.652426 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.652399 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-envoy" pod "7de3942d-307c-4f4c-b291-2b69ad18f472" (UID: "7de3942d-307c-4f4c-b291-2b69ad18f472"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:39.652789 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.652766 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de3942d-307c-4f4c-b291-2b69ad18f472-kube-api-access-t6djp" pod "7de3942d-307c-4f4c-b291-2b69ad18f472" (UID: "7de3942d-307c-4f4c-b291-2b69ad18f472"). InnerVolumeSpecName "kube-api-access-t6djp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:39.771193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.771080 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:39.771193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.771127 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:39.771193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.771144 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6djp\" (UniqueName: \"kubernetes.io/projected/7de3942d-307c-4f4c-b291-2b69ad18f472-kube-api-access-t6djp\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:39.771193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:39.771160 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de3942d-307c-4f4c-b291-2b69ad18f472-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:40.193700 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.193663 5111 generic.go:356] "Generic (PLEG): container finished" podID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerID="42ac199a7bb1fa7a351c74c38ae39ac877081c64bcebb1f417ab7b939396fd8a" exitCode=0 Nov 03 20:19:40.194166 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.193719 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" Nov 03 20:19:40.194166 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.193729 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" event={"ID":"7de3942d-307c-4f4c-b291-2b69ad18f472","Type":"ContainerDied","Data":"42ac199a7bb1fa7a351c74c38ae39ac877081c64bcebb1f417ab7b939396fd8a"} Nov 03 20:19:40.194166 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.193779 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p" event={"ID":"7de3942d-307c-4f4c-b291-2b69ad18f472","Type":"ContainerDied","Data":"2f355898d64850fc4dabec5dfddd21846af1eb0e30a5785290d2dbd1a5a7c643"} Nov 03 20:19:40.194166 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.193809 5111 scope.go:122] "RemoveContainer" containerID="42ac199a7bb1fa7a351c74c38ae39ac877081c64bcebb1f417ab7b939396fd8a" Nov 03 20:19:40.215699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.215589 5111 scope.go:122] "RemoveContainer" containerID="1e8d76fd89609448f8f6a2d4f962288558792de0ccea2dc41e0e394942b9f617" Nov 03 20:19:40.238915 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.238835 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p"] Nov 03 20:19:40.239369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.239342 5111 scope.go:122] "RemoveContainer" containerID="42ac199a7bb1fa7a351c74c38ae39ac877081c64bcebb1f417ab7b939396fd8a" Nov 03 20:19:40.240055 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:19:40.239784 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ac199a7bb1fa7a351c74c38ae39ac877081c64bcebb1f417ab7b939396fd8a\": container with ID starting with 42ac199a7bb1fa7a351c74c38ae39ac877081c64bcebb1f417ab7b939396fd8a not found: ID does not exist" containerID="42ac199a7bb1fa7a351c74c38ae39ac877081c64bcebb1f417ab7b939396fd8a" Nov 03 20:19:40.240055 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.239823 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ac199a7bb1fa7a351c74c38ae39ac877081c64bcebb1f417ab7b939396fd8a"} err="failed to get container status \"42ac199a7bb1fa7a351c74c38ae39ac877081c64bcebb1f417ab7b939396fd8a\": rpc error: code = NotFound desc = could not find container \"42ac199a7bb1fa7a351c74c38ae39ac877081c64bcebb1f417ab7b939396fd8a\": container with ID starting with 42ac199a7bb1fa7a351c74c38ae39ac877081c64bcebb1f417ab7b939396fd8a not found: ID does not exist" Nov 03 20:19:40.240055 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.239848 5111 scope.go:122] "RemoveContainer" containerID="1e8d76fd89609448f8f6a2d4f962288558792de0ccea2dc41e0e394942b9f617" Nov 03 20:19:40.240403 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:19:40.240373 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8d76fd89609448f8f6a2d4f962288558792de0ccea2dc41e0e394942b9f617\": container with ID starting with 1e8d76fd89609448f8f6a2d4f962288558792de0ccea2dc41e0e394942b9f617 not found: ID does not exist" containerID="1e8d76fd89609448f8f6a2d4f962288558792de0ccea2dc41e0e394942b9f617" Nov 03 20:19:40.240500 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.240407 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8d76fd89609448f8f6a2d4f962288558792de0ccea2dc41e0e394942b9f617"} err="failed to get container status \"1e8d76fd89609448f8f6a2d4f962288558792de0ccea2dc41e0e394942b9f617\": rpc error: code = NotFound desc = could not find container \"1e8d76fd89609448f8f6a2d4f962288558792de0ccea2dc41e0e394942b9f617\": container with ID starting with 1e8d76fd89609448f8f6a2d4f962288558792de0ccea2dc41e0e394942b9f617 not found: ID does not exist" Nov 03 20:19:40.247407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.247356 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-q8l8p"] Nov 03 20:19:40.385294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.385211 5111 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:19:40.385294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.385269 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" podUID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:19:40.512605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.507837 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" Nov 03 20:19:40.666475 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.666445 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" Nov 03 20:19:40.670883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.670860 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" Nov 03 20:19:40.676202 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.676174 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" Nov 03 20:19:40.689979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.689949 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21aa1610-bd37-4be6-8ef5-4217c931f3a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21aa1610-bd37-4be6-8ef5-4217c931f3a5-maas-api-openshift-ca\") pod \"21aa1610-bd37-4be6-8ef5-4217c931f3a5\" (UID: \"21aa1610-bd37-4be6-8ef5-4217c931f3a5\") " Nov 03 20:19:40.690140 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.690096 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21aa1610-bd37-4be6-8ef5-4217c931f3a5-kube-api-access-qnps7\" (UniqueName: \"kubernetes.io/projected/21aa1610-bd37-4be6-8ef5-4217c931f3a5-kube-api-access-qnps7\") pod \"21aa1610-bd37-4be6-8ef5-4217c931f3a5\" (UID: \"21aa1610-bd37-4be6-8ef5-4217c931f3a5\") " Nov 03 20:19:40.690928 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.690269 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21aa1610-bd37-4be6-8ef5-4217c931f3a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21aa1610-bd37-4be6-8ef5-4217c931f3a5-maas-api-xks-ca\") pod \"21aa1610-bd37-4be6-8ef5-4217c931f3a5\" (UID: \"21aa1610-bd37-4be6-8ef5-4217c931f3a5\") " Nov 03 20:19:40.690928 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.690642 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21aa1610-bd37-4be6-8ef5-4217c931f3a5-maas-api-openshift-ca" pod "21aa1610-bd37-4be6-8ef5-4217c931f3a5" (UID: "21aa1610-bd37-4be6-8ef5-4217c931f3a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:40.692000 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.691972 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21aa1610-bd37-4be6-8ef5-4217c931f3a5-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:40.692972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.692935 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21aa1610-bd37-4be6-8ef5-4217c931f3a5-maas-api-xks-ca" pod "21aa1610-bd37-4be6-8ef5-4217c931f3a5" (UID: "21aa1610-bd37-4be6-8ef5-4217c931f3a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:19:40.695117 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.694916 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21aa1610-bd37-4be6-8ef5-4217c931f3a5-kube-api-access-qnps7" pod "21aa1610-bd37-4be6-8ef5-4217c931f3a5" (UID: "21aa1610-bd37-4be6-8ef5-4217c931f3a5"). InnerVolumeSpecName "kube-api-access-qnps7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:40.820955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.817358 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25baca52-d88b-4838-be57-1975753451ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25baca52-d88b-4838-be57-1975753451ed-maas-api-xks-ca\") pod \"25baca52-d88b-4838-be57-1975753451ed\" (UID: \"25baca52-d88b-4838-be57-1975753451ed\") " Nov 03 20:19:40.820955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.817440 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e1b3b2-3628-4a69-a57a-60f399cc1038-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e1b3b2-3628-4a69-a57a-60f399cc1038-maas-api-openshift-ca\") pod \"98e1b3b2-3628-4a69-a57a-60f399cc1038\" (UID: \"98e1b3b2-3628-4a69-a57a-60f399cc1038\") " Nov 03 20:19:40.820955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.817474 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-maas-api-openshift-ca\") pod \"386d1427-eb8e-4eba-b1ba-f0bac209dfa9\" (UID: \"386d1427-eb8e-4eba-b1ba-f0bac209dfa9\") " Nov 03 20:19:40.820955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.817566 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25baca52-d88b-4838-be57-1975753451ed-kube-api-access-zvpb6\" (UniqueName: \"kubernetes.io/projected/25baca52-d88b-4838-be57-1975753451ed-kube-api-access-zvpb6\") pod \"25baca52-d88b-4838-be57-1975753451ed\" (UID: \"25baca52-d88b-4838-be57-1975753451ed\") " Nov 03 20:19:40.820955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.817653 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-maas-api-xks-ca\") pod \"386d1427-eb8e-4eba-b1ba-f0bac209dfa9\" (UID: \"386d1427-eb8e-4eba-b1ba-f0bac209dfa9\") " Nov 03 20:19:40.820955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.817758 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25baca52-d88b-4838-be57-1975753451ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25baca52-d88b-4838-be57-1975753451ed-maas-api-openshift-ca\") pod \"25baca52-d88b-4838-be57-1975753451ed\" (UID: \"25baca52-d88b-4838-be57-1975753451ed\") " Nov 03 20:19:40.820955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.817935 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-kube-api-access-npcg8\" (UniqueName: \"kubernetes.io/projected/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-kube-api-access-npcg8\") pod \"386d1427-eb8e-4eba-b1ba-f0bac209dfa9\" (UID: \"386d1427-eb8e-4eba-b1ba-f0bac209dfa9\") " Nov 03 20:19:40.820955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.817987 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e1b3b2-3628-4a69-a57a-60f399cc1038-kube-api-access-wnlp6\" (UniqueName: \"kubernetes.io/projected/98e1b3b2-3628-4a69-a57a-60f399cc1038-kube-api-access-wnlp6\") pod \"98e1b3b2-3628-4a69-a57a-60f399cc1038\" (UID: \"98e1b3b2-3628-4a69-a57a-60f399cc1038\") " Nov 03 20:19:40.820955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.818020 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98e1b3b2-3628-4a69-a57a-60f399cc1038-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e1b3b2-3628-4a69-a57a-60f399cc1038-maas-api-xks-ca\") pod \"98e1b3b2-3628-4a69-a57a-60f399cc1038\" (UID: \"98e1b3b2-3628-4a69-a57a-60f399cc1038\") " Nov 03 20:19:40.820955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.818886 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21aa1610-bd37-4be6-8ef5-4217c931f3a5-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:40.820955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.818911 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnps7\" (UniqueName: \"kubernetes.io/projected/21aa1610-bd37-4be6-8ef5-4217c931f3a5-kube-api-access-qnps7\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:40.822476 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.822170 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-maas-api-openshift-ca" pod "386d1427-eb8e-4eba-b1ba-f0bac209dfa9" (UID: "386d1427-eb8e-4eba-b1ba-f0bac209dfa9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:40.822700 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.822658 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e1b3b2-3628-4a69-a57a-60f399cc1038-maas-api-openshift-ca" pod "98e1b3b2-3628-4a69-a57a-60f399cc1038" (UID: "98e1b3b2-3628-4a69-a57a-60f399cc1038"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:40.823563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.823523 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25baca52-d88b-4838-be57-1975753451ed-maas-api-openshift-ca" pod "25baca52-d88b-4838-be57-1975753451ed" (UID: "25baca52-d88b-4838-be57-1975753451ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:40.831044 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.831000 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-kube-api-access-npcg8" pod "386d1427-eb8e-4eba-b1ba-f0bac209dfa9" (UID: "386d1427-eb8e-4eba-b1ba-f0bac209dfa9"). InnerVolumeSpecName "kube-api-access-npcg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:40.836896 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.836839 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e1b3b2-3628-4a69-a57a-60f399cc1038-kube-api-access-wnlp6" pod "98e1b3b2-3628-4a69-a57a-60f399cc1038" (UID: "98e1b3b2-3628-4a69-a57a-60f399cc1038"). InnerVolumeSpecName "kube-api-access-wnlp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:40.837370 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.837215 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25baca52-d88b-4838-be57-1975753451ed-kube-api-access-zvpb6" pod "25baca52-d88b-4838-be57-1975753451ed" (UID: "25baca52-d88b-4838-be57-1975753451ed"). InnerVolumeSpecName "kube-api-access-zvpb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:40.837370 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.837286 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98e1b3b2-3628-4a69-a57a-60f399cc1038-maas-api-xks-ca" pod "98e1b3b2-3628-4a69-a57a-60f399cc1038" (UID: "98e1b3b2-3628-4a69-a57a-60f399cc1038"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:19:40.837370 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.837337 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-maas-api-xks-ca" pod "386d1427-eb8e-4eba-b1ba-f0bac209dfa9" (UID: "386d1427-eb8e-4eba-b1ba-f0bac209dfa9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:19:40.839291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.839255 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25baca52-d88b-4838-be57-1975753451ed-maas-api-xks-ca" pod "25baca52-d88b-4838-be57-1975753451ed" (UID: "25baca52-d88b-4838-be57-1975753451ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:19:40.928250 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.923249 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npcg8\" (UniqueName: \"kubernetes.io/projected/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-kube-api-access-npcg8\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:40.928250 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.923294 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnlp6\" (UniqueName: \"kubernetes.io/projected/98e1b3b2-3628-4a69-a57a-60f399cc1038-kube-api-access-wnlp6\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:40.928250 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.923311 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e1b3b2-3628-4a69-a57a-60f399cc1038-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:40.928250 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.923325 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25baca52-d88b-4838-be57-1975753451ed-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:40.928250 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.923339 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e1b3b2-3628-4a69-a57a-60f399cc1038-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:40.928250 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.923352 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:40.928250 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.923368 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvpb6\" (UniqueName: \"kubernetes.io/projected/25baca52-d88b-4838-be57-1975753451ed-kube-api-access-zvpb6\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:40.928250 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.923383 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386d1427-eb8e-4eba-b1ba-f0bac209dfa9-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:40.928250 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:40.923395 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25baca52-d88b-4838-be57-1975753451ed-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:41.205063 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:41.205031 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" Nov 03 20:19:41.205519 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:41.205033 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4" event={"ID":"21aa1610-bd37-4be6-8ef5-4217c931f3a5","Type":"ContainerDied","Data":"214ebf6473f1611d43fae66363eab1326c18641e01f8d9ec5de02dd98bb6a660"} Nov 03 20:19:41.205519 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:41.205171 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="214ebf6473f1611d43fae66363eab1326c18641e01f8d9ec5de02dd98bb6a660" Nov 03 20:19:41.206601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:41.206568 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" event={"ID":"25baca52-d88b-4838-be57-1975753451ed","Type":"ContainerDied","Data":"d23fd49eb942add22887b2675ba2faa41559aac56321491dd2d6e3dd5a280e9a"} Nov 03 20:19:41.206722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:41.206628 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d23fd49eb942add22887b2675ba2faa41559aac56321491dd2d6e3dd5a280e9a" Nov 03 20:19:41.206722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:41.206578 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg" Nov 03 20:19:41.210020 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:41.209997 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" Nov 03 20:19:41.210152 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:41.210036 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv" event={"ID":"98e1b3b2-3628-4a69-a57a-60f399cc1038","Type":"ContainerDied","Data":"a65195586b0676b4eeda8c429e3aa2bde46a6c4f19b39a2c970964a4c9181718"} Nov 03 20:19:41.210152 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:41.210147 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a65195586b0676b4eeda8c429e3aa2bde46a6c4f19b39a2c970964a4c9181718" Nov 03 20:19:41.213970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:41.213899 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" event={"ID":"386d1427-eb8e-4eba-b1ba-f0bac209dfa9","Type":"ContainerDied","Data":"296f86736e0f7ccf118c1b81eb1aa0bbc6b577aaf4dab22ed7221ab3d6f8061f"} Nov 03 20:19:41.213970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:41.213942 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="296f86736e0f7ccf118c1b81eb1aa0bbc6b577aaf4dab22ed7221ab3d6f8061f" Nov 03 20:19:41.214099 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:41.213994 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn" Nov 03 20:19:42.337667 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.331706 5111 generic.go:356] "Generic (PLEG): container finished" podID="1a9914b3-5374-41b7-b530-480d2ec5a5ab" containerID="94f4db1a819f549f5639d4a3d09326e27ee12cbcb2d3e2a905ccb0a70a067e99" exitCode=0 Nov 03 20:19:42.427248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.427210 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7de3942d-307c-4f4c-b291-2b69ad18f472" path="/var/lib/kubelet/pods/7de3942d-307c-4f4c-b291-2b69ad18f472/volumes" Nov 03 20:19:42.428054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.427833 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" event={"ID":"1a9914b3-5374-41b7-b530-480d2ec5a5ab","Type":"ContainerDied","Data":"94f4db1a819f549f5639d4a3d09326e27ee12cbcb2d3e2a905ccb0a70a067e99"} Nov 03 20:19:42.532646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.532611 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:42.630988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.622258 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-podinfo\") pod \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " Nov 03 20:19:42.630988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.622438 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-envoy\") pod \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " Nov 03 20:19:42.630988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.622504 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istiod-ca-cert\") pod \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " Nov 03 20:19:42.630988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.622631 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-workload-certs\") pod \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " Nov 03 20:19:42.630988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.622678 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-credential-socket\") pod \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " Nov 03 20:19:42.630988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.622739 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-workload-socket\") pod \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " Nov 03 20:19:42.630988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.622806 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-data\") pod \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " Nov 03 20:19:42.630988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.622895 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-token\" (UniqueName: \"kubernetes.io/projected/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-token\") pod \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " Nov 03 20:19:42.630988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.622963 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9914b3-5374-41b7-b530-480d2ec5a5ab-kube-api-access-fnhtl\" (UniqueName: \"kubernetes.io/projected/1a9914b3-5374-41b7-b530-480d2ec5a5ab-kube-api-access-fnhtl\") pod \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\" (UID: \"1a9914b3-5374-41b7-b530-480d2ec5a5ab\") " Nov 03 20:19:42.633590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.632102 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istiod-ca-cert" pod "1a9914b3-5374-41b7-b530-480d2ec5a5ab" (UID: "1a9914b3-5374-41b7-b530-480d2ec5a5ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:42.633590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.632397 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-workload-socket" pod "1a9914b3-5374-41b7-b530-480d2ec5a5ab" (UID: "1a9914b3-5374-41b7-b530-480d2ec5a5ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:42.633590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.632731 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-credential-socket" pod "1a9914b3-5374-41b7-b530-480d2ec5a5ab" (UID: "1a9914b3-5374-41b7-b530-480d2ec5a5ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:42.633590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.632783 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-workload-certs" pod "1a9914b3-5374-41b7-b530-480d2ec5a5ab" (UID: "1a9914b3-5374-41b7-b530-480d2ec5a5ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:42.633590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.633003 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-data" pod "1a9914b3-5374-41b7-b530-480d2ec5a5ab" (UID: "1a9914b3-5374-41b7-b530-480d2ec5a5ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:42.657009 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.656963 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-envoy" pod "1a9914b3-5374-41b7-b530-480d2ec5a5ab" (UID: "1a9914b3-5374-41b7-b530-480d2ec5a5ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:42.661254 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.661218 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9914b3-5374-41b7-b530-480d2ec5a5ab-kube-api-access-fnhtl" pod "1a9914b3-5374-41b7-b530-480d2ec5a5ab" (UID: "1a9914b3-5374-41b7-b530-480d2ec5a5ab"). InnerVolumeSpecName "kube-api-access-fnhtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:42.667690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.667639 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-podinfo" pod "1a9914b3-5374-41b7-b530-480d2ec5a5ab" (UID: "1a9914b3-5374-41b7-b530-480d2ec5a5ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:19:42.667965 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.667940 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-token" pod "1a9914b3-5374-41b7-b530-480d2ec5a5ab" (UID: "1a9914b3-5374-41b7-b530-480d2ec5a5ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:42.726273 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.726236 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:42.726273 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.726275 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:42.726559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.726291 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:42.726559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.726305 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:42.726559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.726319 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:42.726559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.726334 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:42.726559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.726350 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:42.726559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.726364 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a9914b3-5374-41b7-b530-480d2ec5a5ab-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:42.726559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:42.726380 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnhtl\" (UniqueName: \"kubernetes.io/projected/1a9914b3-5374-41b7-b530-480d2ec5a5ab-kube-api-access-fnhtl\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:43.348667 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:43.348635 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" Nov 03 20:19:43.349168 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:43.348667 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl" event={"ID":"1a9914b3-5374-41b7-b530-480d2ec5a5ab","Type":"ContainerDied","Data":"8293afa0875ef636278eec7cad8af6ecddc331c3daecf3589cdb22e226da757d"} Nov 03 20:19:43.349168 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:43.348744 5111 scope.go:122] "RemoveContainer" containerID="94f4db1a819f549f5639d4a3d09326e27ee12cbcb2d3e2a905ccb0a70a067e99" Nov 03 20:19:43.381490 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:43.381444 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl"] Nov 03 20:19:43.389242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:43.389210 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-t84tl"] Nov 03 20:19:44.227909 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.214957 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a9914b3-5374-41b7-b530-480d2ec5a5ab" path="/var/lib/kubelet/pods/1a9914b3-5374-41b7-b530-480d2ec5a5ab/volumes" Nov 03 20:19:44.432575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.430314 5111 generic.go:356] "Generic (PLEG): container finished" podID="a0da8389-92b0-42da-96cd-4dec2757e02a" containerID="243bd3b731e109ef7d8ee2279d84bb5d97405998fee26922447feeed7744b67b" exitCode=0 Nov 03 20:19:44.432575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.430435 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" event={"ID":"a0da8389-92b0-42da-96cd-4dec2757e02a","Type":"ContainerDied","Data":"243bd3b731e109ef7d8ee2279d84bb5d97405998fee26922447feeed7744b67b"} Nov 03 20:19:44.437045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.434778 5111 generic.go:356] "Generic (PLEG): container finished" podID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerID="d093afa4d10294a4d4661b5d96263fc434fd51f7efeb98e86cc285976a039859" exitCode=0 Nov 03 20:19:44.437045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.434959 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" event={"ID":"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c","Type":"ContainerDied","Data":"d093afa4d10294a4d4661b5d96263fc434fd51f7efeb98e86cc285976a039859"} Nov 03 20:19:44.437045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.435013 5111 scope.go:122] "RemoveContainer" containerID="cb89a204892d6df55eedef7c1985ec05d283c25586c3c8caf7487af583d70821" Nov 03 20:19:44.437824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.437800 5111 generic.go:356] "Generic (PLEG): container finished" podID="8f26fe8b-ab5c-4978-ab59-d163324180ff" containerID="8371bf3a77d805118c08fb37a0d865b00b7e22109a89f5b3d2821cb539b01fb4" exitCode=0 Nov 03 20:19:44.438019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.437996 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" event={"ID":"8f26fe8b-ab5c-4978-ab59-d163324180ff","Type":"ContainerDied","Data":"8371bf3a77d805118c08fb37a0d865b00b7e22109a89f5b3d2821cb539b01fb4"} Nov 03 20:19:44.627600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.627106 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:19:44.729553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.724728 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f26fe8b-ab5c-4978-ab59-d163324180ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f26fe8b-ab5c-4978-ab59-d163324180ff-istiod-ca-cert\") pod \"8f26fe8b-ab5c-4978-ab59-d163324180ff\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " Nov 03 20:19:44.729553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.724845 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-podinfo\") pod \"8f26fe8b-ab5c-4978-ab59-d163324180ff\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " Nov 03 20:19:44.729553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.724892 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-envoy\") pod \"8f26fe8b-ab5c-4978-ab59-d163324180ff\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " Nov 03 20:19:44.729553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.724966 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-workload-certs\") pod \"8f26fe8b-ab5c-4978-ab59-d163324180ff\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " Nov 03 20:19:44.729553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.725112 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f26fe8b-ab5c-4978-ab59-d163324180ff-kube-api-access-2wktk\" (UniqueName: \"kubernetes.io/projected/8f26fe8b-ab5c-4978-ab59-d163324180ff-kube-api-access-2wktk\") pod \"8f26fe8b-ab5c-4978-ab59-d163324180ff\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " Nov 03 20:19:44.729553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.725209 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-data\") pod \"8f26fe8b-ab5c-4978-ab59-d163324180ff\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " Nov 03 20:19:44.729553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.725264 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-credential-socket\") pod \"8f26fe8b-ab5c-4978-ab59-d163324180ff\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " Nov 03 20:19:44.729553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.725308 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-token\" (UniqueName: \"kubernetes.io/projected/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-token\") pod \"8f26fe8b-ab5c-4978-ab59-d163324180ff\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " Nov 03 20:19:44.729553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.725386 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-workload-socket\") pod \"8f26fe8b-ab5c-4978-ab59-d163324180ff\" (UID: \"8f26fe8b-ab5c-4978-ab59-d163324180ff\") " Nov 03 20:19:44.732308 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.732127 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-workload-certs" pod "8f26fe8b-ab5c-4978-ab59-d163324180ff" (UID: "8f26fe8b-ab5c-4978-ab59-d163324180ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:44.733014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.732703 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f26fe8b-ab5c-4978-ab59-d163324180ff-istiod-ca-cert" pod "8f26fe8b-ab5c-4978-ab59-d163324180ff" (UID: "8f26fe8b-ab5c-4978-ab59-d163324180ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:44.763658 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.763187 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-data" pod "8f26fe8b-ab5c-4978-ab59-d163324180ff" (UID: "8f26fe8b-ab5c-4978-ab59-d163324180ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:44.782736 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:19:44.782487 5111 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f272466_fdc2_4521_9f68_782e0cb8da27.slice/crio-3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f272466_fdc2_4521_9f68_782e0cb8da27.slice/crio-conmon-3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:19:44.790314 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:19:44.785306 5111 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f272466_fdc2_4521_9f68_782e0cb8da27.slice/crio-3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f272466_fdc2_4521_9f68_782e0cb8da27.slice/crio-conmon-3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:19:44.792372 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.791795 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-workload-socket" pod "8f26fe8b-ab5c-4978-ab59-d163324180ff" (UID: "8f26fe8b-ab5c-4978-ab59-d163324180ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:44.793355 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.793316 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-credential-socket" pod "8f26fe8b-ab5c-4978-ab59-d163324180ff" (UID: "8f26fe8b-ab5c-4978-ab59-d163324180ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:44.794085 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.794064 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:44.794298 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.794273 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:44.794454 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.794439 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:44.794626 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.794611 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:44.794768 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.794752 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f26fe8b-ab5c-4978-ab59-d163324180ff-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:44.800904 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.800859 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:19:44.818511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.818477 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-token" pod "8f26fe8b-ab5c-4978-ab59-d163324180ff" (UID: "8f26fe8b-ab5c-4978-ab59-d163324180ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:44.819864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.819504 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-envoy" pod "8f26fe8b-ab5c-4978-ab59-d163324180ff" (UID: "8f26fe8b-ab5c-4978-ab59-d163324180ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:44.820015 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.819930 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-podinfo" pod "8f26fe8b-ab5c-4978-ab59-d163324180ff" (UID: "8f26fe8b-ab5c-4978-ab59-d163324180ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:19:44.843096 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.843006 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f26fe8b-ab5c-4978-ab59-d163324180ff-kube-api-access-2wktk" pod "8f26fe8b-ab5c-4978-ab59-d163324180ff" (UID: "8f26fe8b-ab5c-4978-ab59-d163324180ff"). InnerVolumeSpecName "kube-api-access-2wktk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:44.895621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.895503 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0da8389-92b0-42da-96cd-4dec2757e02a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0da8389-92b0-42da-96cd-4dec2757e02a-istiod-ca-cert\") pod \"a0da8389-92b0-42da-96cd-4dec2757e02a\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " Nov 03 20:19:44.895872 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.895849 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-envoy\") pod \"a0da8389-92b0-42da-96cd-4dec2757e02a\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " Nov 03 20:19:44.896434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.896413 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-workload-socket\") pod \"a0da8389-92b0-42da-96cd-4dec2757e02a\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " Nov 03 20:19:44.896978 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.896957 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-token\" (UniqueName: \"kubernetes.io/projected/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-token\") pod \"a0da8389-92b0-42da-96cd-4dec2757e02a\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " Nov 03 20:19:44.899134 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.899114 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0da8389-92b0-42da-96cd-4dec2757e02a-kube-api-access-tpl6s\" (UniqueName: \"kubernetes.io/projected/a0da8389-92b0-42da-96cd-4dec2757e02a-kube-api-access-tpl6s\") pod \"a0da8389-92b0-42da-96cd-4dec2757e02a\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " Nov 03 20:19:44.899632 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.899613 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-data\") pod \"a0da8389-92b0-42da-96cd-4dec2757e02a\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " Nov 03 20:19:44.912753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.912722 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-podinfo\") pod \"a0da8389-92b0-42da-96cd-4dec2757e02a\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " Nov 03 20:19:44.913049 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.913026 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-workload-certs\") pod \"a0da8389-92b0-42da-96cd-4dec2757e02a\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " Nov 03 20:19:44.913822 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.913802 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-credential-socket\") pod \"a0da8389-92b0-42da-96cd-4dec2757e02a\" (UID: \"a0da8389-92b0-42da-96cd-4dec2757e02a\") " Nov 03 20:19:44.915308 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.915284 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:44.915471 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.915454 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:44.915642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.915626 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f26fe8b-ab5c-4978-ab59-d163324180ff-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:44.915747 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.915732 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wktk\" (UniqueName: \"kubernetes.io/projected/8f26fe8b-ab5c-4978-ab59-d163324180ff-kube-api-access-2wktk\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:44.917252 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.896078 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0da8389-92b0-42da-96cd-4dec2757e02a-istiod-ca-cert" pod "a0da8389-92b0-42da-96cd-4dec2757e02a" (UID: "a0da8389-92b0-42da-96cd-4dec2757e02a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:44.917342 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.896874 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-workload-socket" pod "a0da8389-92b0-42da-96cd-4dec2757e02a" (UID: "a0da8389-92b0-42da-96cd-4dec2757e02a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:44.917342 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.913636 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-workload-certs" pod "a0da8389-92b0-42da-96cd-4dec2757e02a" (UID: "a0da8389-92b0-42da-96cd-4dec2757e02a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:44.917342 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.914136 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-credential-socket" pod "a0da8389-92b0-42da-96cd-4dec2757e02a" (UID: "a0da8389-92b0-42da-96cd-4dec2757e02a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:44.917342 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.916680 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-podinfo" pod "a0da8389-92b0-42da-96cd-4dec2757e02a" (UID: "a0da8389-92b0-42da-96cd-4dec2757e02a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:19:44.917342 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.917210 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-data" pod "a0da8389-92b0-42da-96cd-4dec2757e02a" (UID: "a0da8389-92b0-42da-96cd-4dec2757e02a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:44.917675 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.917652 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-envoy" pod "a0da8389-92b0-42da-96cd-4dec2757e02a" (UID: "a0da8389-92b0-42da-96cd-4dec2757e02a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:44.922232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.922201 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-token" pod "a0da8389-92b0-42da-96cd-4dec2757e02a" (UID: "a0da8389-92b0-42da-96cd-4dec2757e02a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:44.924025 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.923987 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0da8389-92b0-42da-96cd-4dec2757e02a-kube-api-access-tpl6s" pod "a0da8389-92b0-42da-96cd-4dec2757e02a" (UID: "a0da8389-92b0-42da-96cd-4dec2757e02a"). InnerVolumeSpecName "kube-api-access-tpl6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:44.967635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:44.967602 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:19:45.005620 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.005590 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:19:45.017704 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.017666 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.017704 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.017701 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.017951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.017711 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpl6s\" (UniqueName: \"kubernetes.io/projected/a0da8389-92b0-42da-96cd-4dec2757e02a-kube-api-access-tpl6s\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.017951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.017723 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.017951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.017734 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.017951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.017747 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.017951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.017760 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.017951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.017769 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0da8389-92b0-42da-96cd-4dec2757e02a-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.017951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.017778 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0da8389-92b0-42da-96cd-4dec2757e02a-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.118957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.118921 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-podinfo\") pod \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " Nov 03 20:19:45.119149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119030 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-envoy\") pod \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " Nov 03 20:19:45.119149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119091 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-data\") pod \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " Nov 03 20:19:45.119255 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119161 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-data\") pod \"8f272466-fdc2-4521-9f68-782e0cb8da27\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " Nov 03 20:19:45.119255 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119215 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-token\" (UniqueName: \"kubernetes.io/projected/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-token\") pod \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " Nov 03 20:19:45.119360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119295 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-envoy\") pod \"8f272466-fdc2-4521-9f68-782e0cb8da27\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " Nov 03 20:19:45.119438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119376 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-workload-certs\") pod \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " Nov 03 20:19:45.119438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119405 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-podinfo\") pod \"8f272466-fdc2-4521-9f68-782e0cb8da27\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " Nov 03 20:19:45.119614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119468 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f272466-fdc2-4521-9f68-782e0cb8da27-kube-api-access-kktmn\" (UniqueName: \"kubernetes.io/projected/8f272466-fdc2-4521-9f68-782e0cb8da27-kube-api-access-kktmn\") pod \"8f272466-fdc2-4521-9f68-782e0cb8da27\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " Nov 03 20:19:45.119614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119470 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-data" pod "8f272466-fdc2-4521-9f68-782e0cb8da27" (UID: "8f272466-fdc2-4521-9f68-782e0cb8da27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:45.119614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119501 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istiod-ca-cert\") pod \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " Nov 03 20:19:45.119614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119607 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-workload-socket\") pod \"8f272466-fdc2-4521-9f68-782e0cb8da27\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " Nov 03 20:19:45.119845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119654 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-workload-certs\") pod \"8f272466-fdc2-4521-9f68-782e0cb8da27\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " Nov 03 20:19:45.119845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119727 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f272466-fdc2-4521-9f68-782e0cb8da27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f272466-fdc2-4521-9f68-782e0cb8da27-istiod-ca-cert\") pod \"8f272466-fdc2-4521-9f68-782e0cb8da27\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " Nov 03 20:19:45.119845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119818 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-credential-socket\") pod \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " Nov 03 20:19:45.120002 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119875 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-credential-socket\") pod \"8f272466-fdc2-4521-9f68-782e0cb8da27\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " Nov 03 20:19:45.120002 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119930 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-token\" (UniqueName: \"kubernetes.io/projected/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-token\") pod \"8f272466-fdc2-4521-9f68-782e0cb8da27\" (UID: \"8f272466-fdc2-4521-9f68-782e0cb8da27\") " Nov 03 20:19:45.120002 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.119978 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-workload-socket\") pod \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " Nov 03 20:19:45.120167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.120011 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-kube-api-access-wm2f7\" (UniqueName: \"kubernetes.io/projected/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-kube-api-access-wm2f7\") pod \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\" (UID: \"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c\") " Nov 03 20:19:45.121130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.120568 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f272466-fdc2-4521-9f68-782e0cb8da27-istiod-ca-cert" pod "8f272466-fdc2-4521-9f68-782e0cb8da27" (UID: "8f272466-fdc2-4521-9f68-782e0cb8da27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:45.121130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.120675 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-credential-socket" pod "8f272466-fdc2-4521-9f68-782e0cb8da27" (UID: "8f272466-fdc2-4521-9f68-782e0cb8da27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:45.121130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.120892 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-credential-socket" pod "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" (UID: "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:45.121130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.121049 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f272466-fdc2-4521-9f68-782e0cb8da27-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.121979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.121272 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istiod-ca-cert" pod "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" (UID: "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:45.121979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.121411 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-workload-socket" pod "8f272466-fdc2-4521-9f68-782e0cb8da27" (UID: "8f272466-fdc2-4521-9f68-782e0cb8da27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:45.122188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.122154 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-workload-certs" pod "8f272466-fdc2-4521-9f68-782e0cb8da27" (UID: "8f272466-fdc2-4521-9f68-782e0cb8da27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:45.122188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.122163 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-workload-socket" pod "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" (UID: "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:45.122285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.122199 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.122285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.122221 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.122285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.122234 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.122433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.122313 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-data" pod "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" (UID: "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:45.122433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.122333 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-envoy" pod "8f272466-fdc2-4521-9f68-782e0cb8da27" (UID: "8f272466-fdc2-4521-9f68-782e0cb8da27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:45.122647 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.122614 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-podinfo" pod "8f272466-fdc2-4521-9f68-782e0cb8da27" (UID: "8f272466-fdc2-4521-9f68-782e0cb8da27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:19:45.122721 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.122676 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-workload-certs" pod "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" (UID: "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:45.124133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.124105 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f272466-fdc2-4521-9f68-782e0cb8da27-kube-api-access-kktmn" pod "8f272466-fdc2-4521-9f68-782e0cb8da27" (UID: "8f272466-fdc2-4521-9f68-782e0cb8da27"). InnerVolumeSpecName "kube-api-access-kktmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:45.124974 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.124952 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-envoy" pod "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" (UID: "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:45.125099 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.125080 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-token" pod "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" (UID: "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:45.125629 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.125596 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-podinfo" pod "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" (UID: "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:19:45.125747 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.125712 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-kube-api-access-wm2f7" pod "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" (UID: "4d2114d1-4f1c-42a1-b0db-02c8f3c5386c"). InnerVolumeSpecName "kube-api-access-wm2f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:45.125845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.125824 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-token" pod "8f272466-fdc2-4521-9f68-782e0cb8da27" (UID: "8f272466-fdc2-4521-9f68-782e0cb8da27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:45.224567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.224468 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.224567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.224510 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.224764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.224527 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.224764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.224588 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.224764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.224603 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kktmn\" (UniqueName: \"kubernetes.io/projected/8f272466-fdc2-4521-9f68-782e0cb8da27-kube-api-access-kktmn\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.224764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.224618 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.224764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.224631 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.224764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.224644 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f272466-fdc2-4521-9f68-782e0cb8da27-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.224764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.224658 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f272466-fdc2-4521-9f68-782e0cb8da27-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.224764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.224672 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.224764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.224688 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm2f7\" (UniqueName: \"kubernetes.io/projected/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-kube-api-access-wm2f7\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.224764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.224702 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.224764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.224715 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.224764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.224729 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:45.449122 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.449091 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" Nov 03 20:19:45.449610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.449097 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9" event={"ID":"8f26fe8b-ab5c-4978-ab59-d163324180ff","Type":"ContainerDied","Data":"afd22a09ed7b78a1155395222cbc3f10e95714a7707ea1f9fcf0c64a43a004f3"} Nov 03 20:19:45.449610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.449228 5111 scope.go:122] "RemoveContainer" containerID="8371bf3a77d805118c08fb37a0d865b00b7e22109a89f5b3d2821cb539b01fb4" Nov 03 20:19:45.451039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.451021 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" Nov 03 20:19:45.451130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.451029 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7" event={"ID":"a0da8389-92b0-42da-96cd-4dec2757e02a","Type":"ContainerDied","Data":"85a680f1e565f8022dfc3dfa97844e39aa5d355345cfbe358568e3848d1c3afd"} Nov 03 20:19:45.458847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.458819 5111 generic.go:356] "Generic (PLEG): container finished" podID="8f272466-fdc2-4521-9f68-782e0cb8da27" containerID="3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39" exitCode=0 Nov 03 20:19:45.458972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.458957 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" Nov 03 20:19:45.459083 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.458949 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" event={"ID":"8f272466-fdc2-4521-9f68-782e0cb8da27","Type":"ContainerDied","Data":"3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39"} Nov 03 20:19:45.459138 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.459112 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l" event={"ID":"8f272466-fdc2-4521-9f68-782e0cb8da27","Type":"ContainerDied","Data":"58c215fb70ab1f5af040f45fe8edaaec4ee2dd2d590efed9b17cc60b2d50778a"} Nov 03 20:19:45.463289 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.463269 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" event={"ID":"4d2114d1-4f1c-42a1-b0db-02c8f3c5386c","Type":"ContainerDied","Data":"9864e8ed8a9441811ee67ee765d0cf05fdef721bbf15c0be08fc24eacdf54b61"} Nov 03 20:19:45.463289 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.463285 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt" Nov 03 20:19:45.464238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.464220 5111 scope.go:122] "RemoveContainer" containerID="243bd3b731e109ef7d8ee2279d84bb5d97405998fee26922447feeed7744b67b" Nov 03 20:19:45.479168 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.479148 5111 scope.go:122] "RemoveContainer" containerID="3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39" Nov 03 20:19:45.483243 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.483218 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9"] Nov 03 20:19:45.491824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.491798 5111 scope.go:122] "RemoveContainer" containerID="3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39" Nov 03 20:19:45.492170 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:19:45.492138 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39\": container with ID starting with 3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39 not found: ID does not exist" containerID="3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39" Nov 03 20:19:45.492249 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.492176 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39"} err="failed to get container status \"3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39\": rpc error: code = NotFound desc = could not find container \"3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39\": container with ID starting with 3a7921db06e418f3e2aab8deb850a730f7ba326a3278497564dd64ae2f927b39 not found: ID does not exist" Nov 03 20:19:45.492249 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.492202 5111 scope.go:122] "RemoveContainer" containerID="d093afa4d10294a4d4661b5d96263fc434fd51f7efeb98e86cc285976a039859" Nov 03 20:19:45.494054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.494034 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-vlkg9"] Nov 03 20:19:45.508899 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.508875 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7"] Nov 03 20:19:45.513686 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.513658 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-wtvs7"] Nov 03 20:19:45.522098 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.522020 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt"] Nov 03 20:19:45.526874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.526855 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-8r7bt"] Nov 03 20:19:45.535579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.535525 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l"] Nov 03 20:19:45.540582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:45.540557 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-8vp8l"] Nov 03 20:19:46.191418 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:46.191372 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" path="/var/lib/kubelet/pods/4d2114d1-4f1c-42a1-b0db-02c8f3c5386c/volumes" Nov 03 20:19:46.192744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:46.192719 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f26fe8b-ab5c-4978-ab59-d163324180ff" path="/var/lib/kubelet/pods/8f26fe8b-ab5c-4978-ab59-d163324180ff/volumes" Nov 03 20:19:46.193332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:46.193317 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f272466-fdc2-4521-9f68-782e0cb8da27" path="/var/lib/kubelet/pods/8f272466-fdc2-4521-9f68-782e0cb8da27/volumes" Nov 03 20:19:46.194146 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:46.194130 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0da8389-92b0-42da-96cd-4dec2757e02a" path="/var/lib/kubelet/pods/a0da8389-92b0-42da-96cd-4dec2757e02a/volumes" Nov 03 20:19:46.329804 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:46.329774 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v"] Nov 03 20:19:46.544580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:46.544481 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf"] Nov 03 20:19:47.141137 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:47.139703 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" Nov 03 20:19:47.141488 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:47.141466 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.788789 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb"] Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790231 5111 cpu_manager.go:414] "containerMap: removing container" podUID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790254 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790326 5111 cpu_manager.go:414] "containerMap: removing container" podUID="386d1427-eb8e-4eba-b1ba-f0bac209dfa9" containerName="revoke-keys" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790334 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386d1427-eb8e-4eba-b1ba-f0bac209dfa9" containerName="revoke-keys" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790358 5111 cpu_manager.go:414] "containerMap: removing container" podUID="1a9914b3-5374-41b7-b530-480d2ec5a5ab" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790367 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9914b3-5374-41b7-b530-480d2ec5a5ab" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790378 5111 cpu_manager.go:414] "containerMap: removing container" podUID="8f26fe8b-ab5c-4978-ab59-d163324180ff" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790385 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f26fe8b-ab5c-4978-ab59-d163324180ff" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790399 5111 cpu_manager.go:414] "containerMap: removing container" podUID="a0da8389-92b0-42da-96cd-4dec2757e02a" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790407 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0da8389-92b0-42da-96cd-4dec2757e02a" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790419 5111 cpu_manager.go:414] "containerMap: removing container" podUID="8f272466-fdc2-4521-9f68-782e0cb8da27" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790432 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f272466-fdc2-4521-9f68-782e0cb8da27" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790443 5111 cpu_manager.go:414] "containerMap: removing container" podUID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790449 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790459 5111 cpu_manager.go:414] "containerMap: removing container" podUID="21aa1610-bd37-4be6-8ef5-4217c931f3a5" containerName="revoke-keys" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790466 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21aa1610-bd37-4be6-8ef5-4217c931f3a5" containerName="revoke-keys" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790509 5111 cpu_manager.go:414] "containerMap: removing container" podUID="25baca52-d88b-4838-be57-1975753451ed" containerName="revoke-keys" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790522 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25baca52-d88b-4838-be57-1975753451ed" containerName="revoke-keys" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790569 5111 cpu_manager.go:414] "containerMap: removing container" podUID="98e1b3b2-3628-4a69-a57a-60f399cc1038" containerName="revoke-keys" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790577 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e1b3b2-3628-4a69-a57a-60f399cc1038" containerName="revoke-keys" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790827 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790858 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="386d1427-eb8e-4eba-b1ba-f0bac209dfa9" containerName="revoke-keys" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790872 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9914b3-5374-41b7-b530-480d2ec5a5ab" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790883 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f26fe8b-ab5c-4978-ab59-d163324180ff" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790891 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0da8389-92b0-42da-96cd-4dec2757e02a" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790900 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790910 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f272466-fdc2-4521-9f68-782e0cb8da27" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790918 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790926 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="21aa1610-bd37-4be6-8ef5-4217c931f3a5" containerName="revoke-keys" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790940 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerName="istio-proxy" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790950 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="25baca52-d88b-4838-be57-1975753451ed" containerName="revoke-keys" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.790960 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e1b3b2-3628-4a69-a57a-60f399cc1038" containerName="revoke-keys" Nov 03 20:19:48.810556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.800161 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb" Nov 03 20:19:48.824597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.823825 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:19:48.824597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.824117 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:19:48.865705 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.865572 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb"] Nov 03 20:19:48.906635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.905345 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a63f401-1dce-4286-ae52-2c570b9fff4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb\" (UID: \"0a63f401-1dce-4286-ae52-2c570b9fff4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb" Nov 03 20:19:48.906879 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.906850 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a63f401-1dce-4286-ae52-2c570b9fff4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb\" (UID: \"0a63f401-1dce-4286-ae52-2c570b9fff4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb" Nov 03 20:19:48.907079 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:48.907060 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5ktg\" (UniqueName: \"kubernetes.io/projected/0a63f401-1dce-4286-ae52-2c570b9fff4b-kube-api-access-v5ktg\") pod \"maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb\" (UID: \"0a63f401-1dce-4286-ae52-2c570b9fff4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb" Nov 03 20:19:49.008961 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:49.008918 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a63f401-1dce-4286-ae52-2c570b9fff4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb\" (UID: \"0a63f401-1dce-4286-ae52-2c570b9fff4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb" Nov 03 20:19:49.009149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:49.009076 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a63f401-1dce-4286-ae52-2c570b9fff4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb\" (UID: \"0a63f401-1dce-4286-ae52-2c570b9fff4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb" Nov 03 20:19:49.009249 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:49.009233 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ktg\" (UniqueName: \"kubernetes.io/projected/0a63f401-1dce-4286-ae52-2c570b9fff4b-kube-api-access-v5ktg\") pod \"maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb\" (UID: \"0a63f401-1dce-4286-ae52-2c570b9fff4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb" Nov 03 20:19:49.009751 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:49.009726 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a63f401-1dce-4286-ae52-2c570b9fff4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb\" (UID: \"0a63f401-1dce-4286-ae52-2c570b9fff4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb" Nov 03 20:19:49.011760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:49.011734 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a63f401-1dce-4286-ae52-2c570b9fff4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb\" (UID: \"0a63f401-1dce-4286-ae52-2c570b9fff4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb" Nov 03 20:19:49.018074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:49.018045 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5ktg\" (UniqueName: \"kubernetes.io/projected/0a63f401-1dce-4286-ae52-2c570b9fff4b-kube-api-access-v5ktg\") pod \"maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb\" (UID: \"0a63f401-1dce-4286-ae52-2c570b9fff4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb" Nov 03 20:19:49.185229 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:49.185198 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb" Nov 03 20:19:49.474886 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:19:49.474854 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a63f401_1dce_4286_ae52_2c570b9fff4b.slice/crio-49caae8ed51a0f62e5cb69828dee9933885b3f79207b5e364a1aa5531d85bb63 WatchSource:0}: Error finding container 49caae8ed51a0f62e5cb69828dee9933885b3f79207b5e364a1aa5531d85bb63: Status 404 returned error can't find the container with id 49caae8ed51a0f62e5cb69828dee9933885b3f79207b5e364a1aa5531d85bb63 Nov 03 20:19:49.475171 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:49.475146 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb"] Nov 03 20:19:49.580021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:49.579984 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb" event={"ID":"0a63f401-1dce-4286-ae52-2c570b9fff4b","Type":"ContainerStarted","Data":"49caae8ed51a0f62e5cb69828dee9933885b3f79207b5e364a1aa5531d85bb63"} Nov 03 20:19:50.591246 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:50.591213 5111 generic.go:356] "Generic (PLEG): container finished" podID="0a63f401-1dce-4286-ae52-2c570b9fff4b" containerID="596b6381aaacf97e444281b1527c932ae1002e90714b8e06ef8d91b1e2b30bc0" exitCode=0 Nov 03 20:19:50.591682 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:50.591260 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb" event={"ID":"0a63f401-1dce-4286-ae52-2c570b9fff4b","Type":"ContainerDied","Data":"596b6381aaacf97e444281b1527c932ae1002e90714b8e06ef8d91b1e2b30bc0"} Nov 03 20:19:51.333390 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:51.333298 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" podUID="96f02f41-3bee-4567-a64f-4ccc94f07b32" containerName="payload-processing" containerID="cri-o://32a1ffc8b6bbc1a1b3aba809c63c0b6275b15f0c607912b5acd7245e5bac36c1" gracePeriod=25 Nov 03 20:19:51.556152 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:51.555966 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" podUID="af81ede7-5819-41d1-94c8-0a297f814fb4" containerName="payload-pre-processing" containerID="cri-o://22e0f0eb5b3fc5c7ca35c7749d2302dbffcb2ce7e2666ee91c4d166e87ee5bab" gracePeriod=25 Nov 03 20:19:51.608600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:51.608521 5111 generic.go:356] "Generic (PLEG): container finished" podID="96f02f41-3bee-4567-a64f-4ccc94f07b32" containerID="32a1ffc8b6bbc1a1b3aba809c63c0b6275b15f0c607912b5acd7245e5bac36c1" exitCode=0 Nov 03 20:19:51.609071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:51.608601 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" event={"ID":"96f02f41-3bee-4567-a64f-4ccc94f07b32","Type":"ContainerDied","Data":"32a1ffc8b6bbc1a1b3aba809c63c0b6275b15f0c607912b5acd7245e5bac36c1"} Nov 03 20:19:51.895584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:51.893189 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" Nov 03 20:19:51.986279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:51.986242 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p"] Nov 03 20:19:51.988151 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:51.988122 5111 cpu_manager.go:414] "containerMap: removing container" podUID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerName="istio-proxy" Nov 03 20:19:51.999606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:51.999572 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2114d1-4f1c-42a1-b0db-02c8f3c5386c" containerName="istio-proxy" Nov 03 20:19:51.999769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:51.999706 5111 cpu_manager.go:414] "containerMap: removing container" podUID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerName="istio-proxy" Nov 03 20:19:51.999769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:51.999720 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de3942d-307c-4f4c-b291-2b69ad18f472" containerName="istio-proxy" Nov 03 20:19:51.999769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:51.999735 5111 cpu_manager.go:414] "containerMap: removing container" podUID="96f02f41-3bee-4567-a64f-4ccc94f07b32" containerName="payload-processing" Nov 03 20:19:51.999769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:51.999743 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96f02f41-3bee-4567-a64f-4ccc94f07b32" containerName="payload-processing" Nov 03 20:19:52.000050 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.000030 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="96f02f41-3bee-4567-a64f-4ccc94f07b32" containerName="payload-processing" Nov 03 20:19:52.016626 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.016598 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p"] Nov 03 20:19:52.016872 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.016705 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" Nov 03 20:19:52.038654 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.038591 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96f02f41-3bee-4567-a64f-4ccc94f07b32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f02f41-3bee-4567-a64f-4ccc94f07b32-plugins-config-volume\") pod \"96f02f41-3bee-4567-a64f-4ccc94f07b32\" (UID: \"96f02f41-3bee-4567-a64f-4ccc94f07b32\") " Nov 03 20:19:52.038854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.038673 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f02f41-3bee-4567-a64f-4ccc94f07b32-kube-api-access-n4dpg\" (UniqueName: \"kubernetes.io/projected/96f02f41-3bee-4567-a64f-4ccc94f07b32-kube-api-access-n4dpg\") pod \"96f02f41-3bee-4567-a64f-4ccc94f07b32\" (UID: \"96f02f41-3bee-4567-a64f-4ccc94f07b32\") " Nov 03 20:19:52.045829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.045752 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f02f41-3bee-4567-a64f-4ccc94f07b32-kube-api-access-n4dpg" pod "96f02f41-3bee-4567-a64f-4ccc94f07b32" (UID: "96f02f41-3bee-4567-a64f-4ccc94f07b32"). InnerVolumeSpecName "kube-api-access-n4dpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:52.047805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.047776 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f02f41-3bee-4567-a64f-4ccc94f07b32-plugins-config-volume" pod "96f02f41-3bee-4567-a64f-4ccc94f07b32" (UID: "96f02f41-3bee-4567-a64f-4ccc94f07b32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:52.079135 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.079105 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb" Nov 03 20:19:52.141261 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.141229 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69d5b785-00c4-40ec-b265-af208e4cc152-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-663013-6w92p\" (UID: \"69d5b785-00c4-40ec-b265-af208e4cc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" Nov 03 20:19:52.141438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.141284 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69d5b785-00c4-40ec-b265-af208e4cc152-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-663013-6w92p\" (UID: \"69d5b785-00c4-40ec-b265-af208e4cc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" Nov 03 20:19:52.141438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.141366 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glfjw\" (UniqueName: \"kubernetes.io/projected/69d5b785-00c4-40ec-b265-af208e4cc152-kube-api-access-glfjw\") pod \"maas-api-revoke-keys-e2e-inf-a-663013-6w92p\" (UID: \"69d5b785-00c4-40ec-b265-af208e4cc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" Nov 03 20:19:52.141638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.141548 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f02f41-3bee-4567-a64f-4ccc94f07b32-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:52.141638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.141571 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4dpg\" (UniqueName: \"kubernetes.io/projected/96f02f41-3bee-4567-a64f-4ccc94f07b32-kube-api-access-n4dpg\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:52.180031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.180005 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" Nov 03 20:19:52.242389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.242335 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a63f401-1dce-4286-ae52-2c570b9fff4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a63f401-1dce-4286-ae52-2c570b9fff4b-maas-api-xks-ca\") pod \"0a63f401-1dce-4286-ae52-2c570b9fff4b\" (UID: \"0a63f401-1dce-4286-ae52-2c570b9fff4b\") " Nov 03 20:19:52.242633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.242407 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a63f401-1dce-4286-ae52-2c570b9fff4b-kube-api-access-v5ktg\" (UniqueName: \"kubernetes.io/projected/0a63f401-1dce-4286-ae52-2c570b9fff4b-kube-api-access-v5ktg\") pod \"0a63f401-1dce-4286-ae52-2c570b9fff4b\" (UID: \"0a63f401-1dce-4286-ae52-2c570b9fff4b\") " Nov 03 20:19:52.242719 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.242681 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a63f401-1dce-4286-ae52-2c570b9fff4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a63f401-1dce-4286-ae52-2c570b9fff4b-maas-api-openshift-ca\") pod \"0a63f401-1dce-4286-ae52-2c570b9fff4b\" (UID: \"0a63f401-1dce-4286-ae52-2c570b9fff4b\") " Nov 03 20:19:52.243265 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.243238 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69d5b785-00c4-40ec-b265-af208e4cc152-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-663013-6w92p\" (UID: \"69d5b785-00c4-40ec-b265-af208e4cc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" Nov 03 20:19:52.243384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.243304 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69d5b785-00c4-40ec-b265-af208e4cc152-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-663013-6w92p\" (UID: \"69d5b785-00c4-40ec-b265-af208e4cc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" Nov 03 20:19:52.243384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.243338 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glfjw\" (UniqueName: \"kubernetes.io/projected/69d5b785-00c4-40ec-b265-af208e4cc152-kube-api-access-glfjw\") pod \"maas-api-revoke-keys-e2e-inf-a-663013-6w92p\" (UID: \"69d5b785-00c4-40ec-b265-af208e4cc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" Nov 03 20:19:52.244351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.244328 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a63f401-1dce-4286-ae52-2c570b9fff4b-maas-api-openshift-ca" pod "0a63f401-1dce-4286-ae52-2c570b9fff4b" (UID: "0a63f401-1dce-4286-ae52-2c570b9fff4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:52.244650 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.244621 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69d5b785-00c4-40ec-b265-af208e4cc152-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-663013-6w92p\" (UID: \"69d5b785-00c4-40ec-b265-af208e4cc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" Nov 03 20:19:52.246242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.246211 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a63f401-1dce-4286-ae52-2c570b9fff4b-maas-api-xks-ca" pod "0a63f401-1dce-4286-ae52-2c570b9fff4b" (UID: "0a63f401-1dce-4286-ae52-2c570b9fff4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:19:52.246940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.246912 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a63f401-1dce-4286-ae52-2c570b9fff4b-kube-api-access-v5ktg" pod "0a63f401-1dce-4286-ae52-2c570b9fff4b" (UID: "0a63f401-1dce-4286-ae52-2c570b9fff4b"). InnerVolumeSpecName "kube-api-access-v5ktg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:52.247283 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.247235 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69d5b785-00c4-40ec-b265-af208e4cc152-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-663013-6w92p\" (UID: \"69d5b785-00c4-40ec-b265-af208e4cc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" Nov 03 20:19:52.252790 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.252755 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glfjw\" (UniqueName: \"kubernetes.io/projected/69d5b785-00c4-40ec-b265-af208e4cc152-kube-api-access-glfjw\") pod \"maas-api-revoke-keys-e2e-inf-a-663013-6w92p\" (UID: \"69d5b785-00c4-40ec-b265-af208e4cc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" Nov 03 20:19:52.402837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.396693 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af81ede7-5819-41d1-94c8-0a297f814fb4-kube-api-access-k565z\" (UniqueName: \"kubernetes.io/projected/af81ede7-5819-41d1-94c8-0a297f814fb4-kube-api-access-k565z\") pod \"af81ede7-5819-41d1-94c8-0a297f814fb4\" (UID: \"af81ede7-5819-41d1-94c8-0a297f814fb4\") " Nov 03 20:19:52.402837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.396900 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af81ede7-5819-41d1-94c8-0a297f814fb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af81ede7-5819-41d1-94c8-0a297f814fb4-plugins-config-volume\") pod \"af81ede7-5819-41d1-94c8-0a297f814fb4\" (UID: \"af81ede7-5819-41d1-94c8-0a297f814fb4\") " Nov 03 20:19:52.402837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.399101 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" Nov 03 20:19:52.419437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.415511 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a63f401-1dce-4286-ae52-2c570b9fff4b-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:52.419437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.415577 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a63f401-1dce-4286-ae52-2c570b9fff4b-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:52.419437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.415593 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5ktg\" (UniqueName: \"kubernetes.io/projected/0a63f401-1dce-4286-ae52-2c570b9fff4b-kube-api-access-v5ktg\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:52.421100 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.421024 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af81ede7-5819-41d1-94c8-0a297f814fb4-plugins-config-volume" pod "af81ede7-5819-41d1-94c8-0a297f814fb4" (UID: "af81ede7-5819-41d1-94c8-0a297f814fb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:52.427332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.425274 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af81ede7-5819-41d1-94c8-0a297f814fb4-kube-api-access-k565z" pod "af81ede7-5819-41d1-94c8-0a297f814fb4" (UID: "af81ede7-5819-41d1-94c8-0a297f814fb4"). InnerVolumeSpecName "kube-api-access-k565z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:52.517456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.517367 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k565z\" (UniqueName: \"kubernetes.io/projected/af81ede7-5819-41d1-94c8-0a297f814fb4-kube-api-access-k565z\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:52.517456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.517404 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af81ede7-5819-41d1-94c8-0a297f814fb4-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:52.620911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.620868 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" event={"ID":"96f02f41-3bee-4567-a64f-4ccc94f07b32","Type":"ContainerDied","Data":"ce570a9a44bd670ac5ceba506058f86f9fed493812f04c3c0af3f975033c26f6"} Nov 03 20:19:52.621428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.620947 5111 scope.go:122] "RemoveContainer" containerID="32a1ffc8b6bbc1a1b3aba809c63c0b6275b15f0c607912b5acd7245e5bac36c1" Nov 03 20:19:52.621428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.620880 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v" Nov 03 20:19:52.623631 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.623608 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb" Nov 03 20:19:52.623792 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.623769 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb" event={"ID":"0a63f401-1dce-4286-ae52-2c570b9fff4b","Type":"ContainerDied","Data":"49caae8ed51a0f62e5cb69828dee9933885b3f79207b5e364a1aa5531d85bb63"} Nov 03 20:19:52.623882 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.623821 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49caae8ed51a0f62e5cb69828dee9933885b3f79207b5e364a1aa5531d85bb63" Nov 03 20:19:52.625887 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.625860 5111 generic.go:356] "Generic (PLEG): container finished" podID="af81ede7-5819-41d1-94c8-0a297f814fb4" containerID="22e0f0eb5b3fc5c7ca35c7749d2302dbffcb2ce7e2666ee91c4d166e87ee5bab" exitCode=0 Nov 03 20:19:52.626005 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.625891 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" event={"ID":"af81ede7-5819-41d1-94c8-0a297f814fb4","Type":"ContainerDied","Data":"22e0f0eb5b3fc5c7ca35c7749d2302dbffcb2ce7e2666ee91c4d166e87ee5bab"} Nov 03 20:19:52.626005 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.625968 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" event={"ID":"af81ede7-5819-41d1-94c8-0a297f814fb4","Type":"ContainerDied","Data":"7fe6b3255f126490823fcd166070a20221a60fb20e4ef7d97c516f552e112806"} Nov 03 20:19:52.626005 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.625982 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf" Nov 03 20:19:52.643301 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.643272 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v"] Nov 03 20:19:52.648019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.647997 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f817b0-759b8969db-rs54v"] Nov 03 20:19:52.648084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.648074 5111 scope.go:122] "RemoveContainer" containerID="22e0f0eb5b3fc5c7ca35c7749d2302dbffcb2ce7e2666ee91c4d166e87ee5bab" Nov 03 20:19:52.661833 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.661779 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf"] Nov 03 20:19:52.662316 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.662296 5111 scope.go:122] "RemoveContainer" containerID="22e0f0eb5b3fc5c7ca35c7749d2302dbffcb2ce7e2666ee91c4d166e87ee5bab" Nov 03 20:19:52.662714 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:19:52.662691 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e0f0eb5b3fc5c7ca35c7749d2302dbffcb2ce7e2666ee91c4d166e87ee5bab\": container with ID starting with 22e0f0eb5b3fc5c7ca35c7749d2302dbffcb2ce7e2666ee91c4d166e87ee5bab not found: ID does not exist" containerID="22e0f0eb5b3fc5c7ca35c7749d2302dbffcb2ce7e2666ee91c4d166e87ee5bab" Nov 03 20:19:52.662787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.662720 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e0f0eb5b3fc5c7ca35c7749d2302dbffcb2ce7e2666ee91c4d166e87ee5bab"} err="failed to get container status \"22e0f0eb5b3fc5c7ca35c7749d2302dbffcb2ce7e2666ee91c4d166e87ee5bab\": rpc error: code = NotFound desc = could not find container \"22e0f0eb5b3fc5c7ca35c7749d2302dbffcb2ce7e2666ee91c4d166e87ee5bab\": container with ID starting with 22e0f0eb5b3fc5c7ca35c7749d2302dbffcb2ce7e2666ee91c4d166e87ee5bab not found: ID does not exist" Nov 03 20:19:52.666623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.666590 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2c6c50-68444fbcd7-kgqvf"] Nov 03 20:19:52.722082 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:52.722046 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p"] Nov 03 20:19:52.722213 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:19:52.722091 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d5b785_00c4_40ec_b265_af208e4cc152.slice/crio-7ded2b09cf245abb638fd951600c903ba64ed5e33062308e7b1813956243b912 WatchSource:0}: Error finding container 7ded2b09cf245abb638fd951600c903ba64ed5e33062308e7b1813956243b912: Status 404 returned error can't find the container with id 7ded2b09cf245abb638fd951600c903ba64ed5e33062308e7b1813956243b912 Nov 03 20:19:53.639118 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:53.639074 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" event={"ID":"69d5b785-00c4-40ec-b265-af208e4cc152","Type":"ContainerStarted","Data":"4c5a39801ee00b39bbcd95b3344fe6d6f501b98709ed0ad1a88a6499705305da"} Nov 03 20:19:53.639630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:53.639146 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" event={"ID":"69d5b785-00c4-40ec-b265-af208e4cc152","Type":"ContainerStarted","Data":"7ded2b09cf245abb638fd951600c903ba64ed5e33062308e7b1813956243b912"} Nov 03 20:19:53.656653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:53.656595 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" podStartSLOduration=2.656578189 podStartE2EDuration="2.656578189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:53.654577261 +0000 UTC m=+2423.616955697" watchObservedRunningTime="2026-11-03 20:19:53.656578189 +0000 UTC m=+2423.618956628" Nov 03 20:19:54.212896 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:54.212862 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96f02f41-3bee-4567-a64f-4ccc94f07b32" path="/var/lib/kubelet/pods/96f02f41-3bee-4567-a64f-4ccc94f07b32/volumes" Nov 03 20:19:54.213652 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:54.213519 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af81ede7-5819-41d1-94c8-0a297f814fb4" path="/var/lib/kubelet/pods/af81ede7-5819-41d1-94c8-0a297f814fb4/volumes" Nov 03 20:19:54.685580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:54.683192 5111 generic.go:356] "Generic (PLEG): container finished" podID="69d5b785-00c4-40ec-b265-af208e4cc152" containerID="4c5a39801ee00b39bbcd95b3344fe6d6f501b98709ed0ad1a88a6499705305da" exitCode=0 Nov 03 20:19:54.685580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:54.683592 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" event={"ID":"69d5b785-00c4-40ec-b265-af208e4cc152","Type":"ContainerDied","Data":"4c5a39801ee00b39bbcd95b3344fe6d6f501b98709ed0ad1a88a6499705305da"} Nov 03 20:19:55.416435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:55.411298 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv"] Nov 03 20:19:55.418905 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:55.418854 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2c6c50-kwbtv"] Nov 03 20:19:55.907798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:55.907771 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" Nov 03 20:19:56.066074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.061235 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d5b785-00c4-40ec-b265-af208e4cc152-kube-api-access-glfjw\" (UniqueName: \"kubernetes.io/projected/69d5b785-00c4-40ec-b265-af208e4cc152-kube-api-access-glfjw\") pod \"69d5b785-00c4-40ec-b265-af208e4cc152\" (UID: \"69d5b785-00c4-40ec-b265-af208e4cc152\") " Nov 03 20:19:56.066074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.061329 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69d5b785-00c4-40ec-b265-af208e4cc152-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69d5b785-00c4-40ec-b265-af208e4cc152-maas-api-xks-ca\") pod \"69d5b785-00c4-40ec-b265-af208e4cc152\" (UID: \"69d5b785-00c4-40ec-b265-af208e4cc152\") " Nov 03 20:19:56.066074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.061715 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d5b785-00c4-40ec-b265-af208e4cc152-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69d5b785-00c4-40ec-b265-af208e4cc152-maas-api-openshift-ca\") pod \"69d5b785-00c4-40ec-b265-af208e4cc152\" (UID: \"69d5b785-00c4-40ec-b265-af208e4cc152\") " Nov 03 20:19:56.066074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.062807 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d5b785-00c4-40ec-b265-af208e4cc152-maas-api-openshift-ca" pod "69d5b785-00c4-40ec-b265-af208e4cc152" (UID: "69d5b785-00c4-40ec-b265-af208e4cc152"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:56.074869 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.074829 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d5b785-00c4-40ec-b265-af208e4cc152-kube-api-access-glfjw" pod "69d5b785-00c4-40ec-b265-af208e4cc152" (UID: "69d5b785-00c4-40ec-b265-af208e4cc152"). InnerVolumeSpecName "kube-api-access-glfjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:56.111300 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.110622 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69d5b785-00c4-40ec-b265-af208e4cc152-maas-api-xks-ca" pod "69d5b785-00c4-40ec-b265-af208e4cc152" (UID: "69d5b785-00c4-40ec-b265-af208e4cc152"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:19:56.125726 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.125697 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66"] Nov 03 20:19:56.125932 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.125904 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" podUID="a9c13b0c-0fbf-47bd-a1e2-602608404ea3" containerName="maas-api" containerID="cri-o://24230e91165f14630a9115f1c999afa5b3f5d77f48a6e993fdac9032cb60b5da" gracePeriod=30 Nov 03 20:19:56.164549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.164497 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69d5b785-00c4-40ec-b265-af208e4cc152-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:56.164716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.164555 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glfjw\" (UniqueName: \"kubernetes.io/projected/69d5b785-00c4-40ec-b265-af208e4cc152-kube-api-access-glfjw\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:56.164716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.164573 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69d5b785-00c4-40ec-b265-af208e4cc152-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:56.198495 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.198447 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e1b3b2-3628-4a69-a57a-60f399cc1038" path="/var/lib/kubelet/pods/98e1b3b2-3628-4a69-a57a-60f399cc1038/volumes" Nov 03 20:19:56.686284 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.685964 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg"] Nov 03 20:19:56.695260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.695200 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f817b0-zwcqg"] Nov 03 20:19:56.710891 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.710853 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" event={"ID":"69d5b785-00c4-40ec-b265-af208e4cc152","Type":"ContainerDied","Data":"7ded2b09cf245abb638fd951600c903ba64ed5e33062308e7b1813956243b912"} Nov 03 20:19:56.711065 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.710910 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ded2b09cf245abb638fd951600c903ba64ed5e33062308e7b1813956243b912" Nov 03 20:19:56.711065 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.711015 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p" Nov 03 20:19:56.713727 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.713700 5111 generic.go:356] "Generic (PLEG): container finished" podID="a9c13b0c-0fbf-47bd-a1e2-602608404ea3" containerID="24230e91165f14630a9115f1c999afa5b3f5d77f48a6e993fdac9032cb60b5da" exitCode=0 Nov 03 20:19:56.713848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.713797 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" event={"ID":"a9c13b0c-0fbf-47bd-a1e2-602608404ea3","Type":"ContainerDied","Data":"24230e91165f14630a9115f1c999afa5b3f5d77f48a6e993fdac9032cb60b5da"} Nov 03 20:19:56.742431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.742406 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:19:56.870220 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.870176 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-kube-api-access-wlp5r\" (UniqueName: \"kubernetes.io/projected/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-kube-api-access-wlp5r\") pod \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\" (UID: \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\") " Nov 03 20:19:56.870403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.870259 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-maas-api-tls\") pod \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\" (UID: \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\") " Nov 03 20:19:56.870403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.870319 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-metrics-certs\") pod \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\" (UID: \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\") " Nov 03 20:19:56.870519 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.870437 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-openshift-service-ca\") pod \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\" (UID: \"a9c13b0c-0fbf-47bd-a1e2-602608404ea3\") " Nov 03 20:19:56.871162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.871129 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-openshift-service-ca" pod "a9c13b0c-0fbf-47bd-a1e2-602608404ea3" (UID: "a9c13b0c-0fbf-47bd-a1e2-602608404ea3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:56.871474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.871426 5111 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-openshift-service-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:56.906431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.888487 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-metrics-certs" pod "a9c13b0c-0fbf-47bd-a1e2-602608404ea3" (UID: "a9c13b0c-0fbf-47bd-a1e2-602608404ea3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:19:56.913880 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.913836 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-kube-api-access-wlp5r" pod "a9c13b0c-0fbf-47bd-a1e2-602608404ea3" (UID: "a9c13b0c-0fbf-47bd-a1e2-602608404ea3"). InnerVolumeSpecName "kube-api-access-wlp5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:56.925848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.925699 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-maas-api-tls" pod "a9c13b0c-0fbf-47bd-a1e2-602608404ea3" (UID: "a9c13b0c-0fbf-47bd-a1e2-602608404ea3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:19:56.974065 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.973963 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlp5r\" (UniqueName: \"kubernetes.io/projected/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-kube-api-access-wlp5r\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:56.974065 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.973999 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-maas-api-tls\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:56.974065 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:56.974018 5111 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c13b0c-0fbf-47bd-a1e2-602608404ea3-metrics-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:57.725199 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:57.725168 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" Nov 03 20:19:57.725379 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:57.725212 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66" event={"ID":"a9c13b0c-0fbf-47bd-a1e2-602608404ea3","Type":"ContainerDied","Data":"d09ec808269f79a8f934e31fe353294ff049f72b3b59d6022c13eba5b77c4284"} Nov 03 20:19:57.725379 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:57.725279 5111 scope.go:122] "RemoveContainer" containerID="24230e91165f14630a9115f1c999afa5b3f5d77f48a6e993fdac9032cb60b5da" Nov 03 20:19:57.751709 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:57.751674 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66"] Nov 03 20:19:57.756312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:57.756289 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f817b0-7d9d679bf5-5wn66"] Nov 03 20:19:58.188855 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:58.188816 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25baca52-d88b-4838-be57-1975753451ed" path="/var/lib/kubelet/pods/25baca52-d88b-4838-be57-1975753451ed/volumes" Nov 03 20:19:58.189546 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:19:58.189516 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c13b0c-0fbf-47bd-a1e2-602608404ea3" path="/var/lib/kubelet/pods/a9c13b0c-0fbf-47bd-a1e2-602608404ea3/volumes" Nov 03 20:20:00.987680 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:00.987594 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb"] Nov 03 20:20:01.006392 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:01.006342 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2092ba-5ljnb"] Nov 03 20:20:02.217514 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:02.217468 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a63f401-1dce-4286-ae52-2c570b9fff4b" path="/var/lib/kubelet/pods/0a63f401-1dce-4286-ae52-2c570b9fff4b/volumes" Nov 03 20:20:02.701554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:02.699877 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f"] Nov 03 20:20:02.701554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:02.700143 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" podUID="30c48c95-5c93-4887-86bb-8e2b76c120bf" containerName="istio-proxy" containerID="cri-o://958d342d9e3d0f5b764ac4389a7c94be62f916c4d6365d2c92ae7838f972db26" gracePeriod=30 Nov 03 20:20:02.710970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:02.710937 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j"] Nov 03 20:20:02.711226 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:02.711193 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" podUID="226bf833-ba6d-4c44-91bd-015817c60b68" containerName="istio-proxy" containerID="cri-o://f6248dfd3770b44252cc97107410e9e9182c70851aaaaba6871bb8810d841edb" gracePeriod=30 Nov 03 20:20:02.721105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:02.721075 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc"] Nov 03 20:20:02.722135 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:02.721323 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" podUID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" containerName="istio-proxy" containerID="cri-o://00327e7941e5036b29f28150fa92dd0818abad17290fe7f20147fe7926729a0c" gracePeriod=30 Nov 03 20:20:04.235185 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.235148 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck"] Nov 03 20:20:04.236712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.235364 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" podUID="a3e211eb-7ac1-4136-b8d0-65211bb869f6" containerName="istio-proxy" containerID="cri-o://846b3c01b9cb7f1dac233ae831f1943fad718283e588e39e894e632fec55f557" gracePeriod=30 Nov 03 20:20:04.614825 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.614742 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js"] Nov 03 20:20:04.615674 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.615654 5111 cpu_manager.go:414] "containerMap: removing container" podUID="69d5b785-00c4-40ec-b265-af208e4cc152" containerName="revoke-keys" Nov 03 20:20:04.615674 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.615674 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d5b785-00c4-40ec-b265-af208e4cc152" containerName="revoke-keys" Nov 03 20:20:04.615940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.615708 5111 cpu_manager.go:414] "containerMap: removing container" podUID="a9c13b0c-0fbf-47bd-a1e2-602608404ea3" containerName="maas-api" Nov 03 20:20:04.615940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.615716 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c13b0c-0fbf-47bd-a1e2-602608404ea3" containerName="maas-api" Nov 03 20:20:04.615940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.615756 5111 cpu_manager.go:414] "containerMap: removing container" podUID="0a63f401-1dce-4286-ae52-2c570b9fff4b" containerName="revoke-keys" Nov 03 20:20:04.615940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.615766 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a63f401-1dce-4286-ae52-2c570b9fff4b" containerName="revoke-keys" Nov 03 20:20:04.615940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.615817 5111 cpu_manager.go:414] "containerMap: removing container" podUID="af81ede7-5819-41d1-94c8-0a297f814fb4" containerName="payload-pre-processing" Nov 03 20:20:04.615940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.615825 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af81ede7-5819-41d1-94c8-0a297f814fb4" containerName="payload-pre-processing" Nov 03 20:20:04.616900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.616033 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d5b785-00c4-40ec-b265-af208e4cc152" containerName="revoke-keys" Nov 03 20:20:04.616900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.616053 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c13b0c-0fbf-47bd-a1e2-602608404ea3" containerName="maas-api" Nov 03 20:20:04.616900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.616069 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a63f401-1dce-4286-ae52-2c570b9fff4b" containerName="revoke-keys" Nov 03 20:20:04.616900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.616085 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="af81ede7-5819-41d1-94c8-0a297f814fb4" containerName="payload-pre-processing" Nov 03 20:20:04.624193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.624158 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" Nov 03 20:20:04.627968 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.627929 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:20:04.628223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.628199 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:20:04.631314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.631268 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js"] Nov 03 20:20:04.725958 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.725928 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9csx2\" (UniqueName: \"kubernetes.io/projected/7ef63c4d-f39d-4623-869c-26983ba16e10-kube-api-access-9csx2\") pod \"maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js\" (UID: \"7ef63c4d-f39d-4623-869c-26983ba16e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" Nov 03 20:20:04.726159 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.725967 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef63c4d-f39d-4623-869c-26983ba16e10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js\" (UID: \"7ef63c4d-f39d-4623-869c-26983ba16e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" Nov 03 20:20:04.726159 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.726099 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef63c4d-f39d-4623-869c-26983ba16e10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js\" (UID: \"7ef63c4d-f39d-4623-869c-26983ba16e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" Nov 03 20:20:04.828041 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.828001 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9csx2\" (UniqueName: \"kubernetes.io/projected/7ef63c4d-f39d-4623-869c-26983ba16e10-kube-api-access-9csx2\") pod \"maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js\" (UID: \"7ef63c4d-f39d-4623-869c-26983ba16e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" Nov 03 20:20:04.828210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.828046 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef63c4d-f39d-4623-869c-26983ba16e10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js\" (UID: \"7ef63c4d-f39d-4623-869c-26983ba16e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" Nov 03 20:20:04.828210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.828093 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef63c4d-f39d-4623-869c-26983ba16e10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js\" (UID: \"7ef63c4d-f39d-4623-869c-26983ba16e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" Nov 03 20:20:04.828747 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.828723 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef63c4d-f39d-4623-869c-26983ba16e10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js\" (UID: \"7ef63c4d-f39d-4623-869c-26983ba16e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" Nov 03 20:20:04.830686 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.830667 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef63c4d-f39d-4623-869c-26983ba16e10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js\" (UID: \"7ef63c4d-f39d-4623-869c-26983ba16e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" Nov 03 20:20:04.836147 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.836127 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9csx2\" (UniqueName: \"kubernetes.io/projected/7ef63c4d-f39d-4623-869c-26983ba16e10-kube-api-access-9csx2\") pod \"maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js\" (UID: \"7ef63c4d-f39d-4623-869c-26983ba16e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" Nov 03 20:20:04.953058 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:04.953018 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" Nov 03 20:20:05.102767 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:20:05.102720 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:20:05.242192 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:05.242160 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js"] Nov 03 20:20:05.242761 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:20:05.242647 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef63c4d_f39d_4623_869c_26983ba16e10.slice/crio-f8777905b3301790491c2acb9f3bb636c93756cc926bbafab75d6fb3b998e9a8 WatchSource:0}: Error finding container f8777905b3301790491c2acb9f3bb636c93756cc926bbafab75d6fb3b998e9a8: Status 404 returned error can't find the container with id f8777905b3301790491c2acb9f3bb636c93756cc926bbafab75d6fb3b998e9a8 Nov 03 20:20:05.942938 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:05.942856 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" event={"ID":"7ef63c4d-f39d-4623-869c-26983ba16e10","Type":"ContainerStarted","Data":"cc28ec4020cd7a802f6297f6cd62c479bffcb36fdead0fd043f34c66bfef511d"} Nov 03 20:20:05.942938 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:05.942911 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" event={"ID":"7ef63c4d-f39d-4623-869c-26983ba16e10","Type":"ContainerStarted","Data":"f8777905b3301790491c2acb9f3bb636c93756cc926bbafab75d6fb3b998e9a8"} Nov 03 20:20:05.959718 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:05.959670 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" podStartSLOduration=1.959656801 podStartE2EDuration="1.959656801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:05.958228292 +0000 UTC m=+2435.920606735" watchObservedRunningTime="2026-11-03 20:20:05.959656801 +0000 UTC m=+2435.922035239" Nov 03 20:20:06.953045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:06.953006 5111 generic.go:356] "Generic (PLEG): container finished" podID="7ef63c4d-f39d-4623-869c-26983ba16e10" containerID="cc28ec4020cd7a802f6297f6cd62c479bffcb36fdead0fd043f34c66bfef511d" exitCode=0 Nov 03 20:20:06.953510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:06.953091 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" event={"ID":"7ef63c4d-f39d-4623-869c-26983ba16e10","Type":"ContainerDied","Data":"cc28ec4020cd7a802f6297f6cd62c479bffcb36fdead0fd043f34c66bfef511d"} Nov 03 20:20:07.968184 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:07.967356 5111 generic.go:356] "Generic (PLEG): container finished" podID="226bf833-ba6d-4c44-91bd-015817c60b68" containerID="f6248dfd3770b44252cc97107410e9e9182c70851aaaaba6871bb8810d841edb" exitCode=0 Nov 03 20:20:07.968184 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:07.967431 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" event={"ID":"226bf833-ba6d-4c44-91bd-015817c60b68","Type":"ContainerDied","Data":"f6248dfd3770b44252cc97107410e9e9182c70851aaaaba6871bb8810d841edb"} Nov 03 20:20:07.972569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:07.969500 5111 generic.go:356] "Generic (PLEG): container finished" podID="30c48c95-5c93-4887-86bb-8e2b76c120bf" containerID="958d342d9e3d0f5b764ac4389a7c94be62f916c4d6365d2c92ae7838f972db26" exitCode=0 Nov 03 20:20:07.972569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:07.969625 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" event={"ID":"30c48c95-5c93-4887-86bb-8e2b76c120bf","Type":"ContainerDied","Data":"958d342d9e3d0f5b764ac4389a7c94be62f916c4d6365d2c92ae7838f972db26"} Nov 03 20:20:07.972569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:07.972034 5111 generic.go:356] "Generic (PLEG): container finished" podID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" containerID="00327e7941e5036b29f28150fa92dd0818abad17290fe7f20147fe7926729a0c" exitCode=0 Nov 03 20:20:07.972569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:07.972329 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" event={"ID":"c58995fe-f4f0-48be-b3a0-03afb09e2ad3","Type":"ContainerDied","Data":"00327e7941e5036b29f28150fa92dd0818abad17290fe7f20147fe7926729a0c"} Nov 03 20:20:07.972569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:07.972370 5111 scope.go:122] "RemoveContainer" containerID="5cf1745f31dfecbb15e41cb1df7f5552ff9f2c672d0a4a0b36287f638eee13ec" Nov 03 20:20:08.106203 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.106048 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:20:08.109682 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.109612 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:20:08.202272 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.201895 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:20:08.242230 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.242138 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-podinfo\") pod \"30c48c95-5c93-4887-86bb-8e2b76c120bf\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " Nov 03 20:20:08.242518 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.242494 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-envoy\") pod \"30c48c95-5c93-4887-86bb-8e2b76c120bf\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " Nov 03 20:20:08.242760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.242741 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-istio-data\") pod \"226bf833-ba6d-4c44-91bd-015817c60b68\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " Nov 03 20:20:08.243228 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.243127 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-istio-data" pod "226bf833-ba6d-4c44-91bd-015817c60b68" (UID: "226bf833-ba6d-4c44-91bd-015817c60b68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:08.243500 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.243463 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-workload-socket" pod "30c48c95-5c93-4887-86bb-8e2b76c120bf" (UID: "30c48c95-5c93-4887-86bb-8e2b76c120bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:08.243638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.243545 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-workload-socket\") pod \"30c48c95-5c93-4887-86bb-8e2b76c120bf\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " Nov 03 20:20:08.243717 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.243638 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c48c95-5c93-4887-86bb-8e2b76c120bf-kube-api-access-8x7cl\" (UniqueName: \"kubernetes.io/projected/30c48c95-5c93-4887-86bb-8e2b76c120bf-kube-api-access-8x7cl\") pod \"30c48c95-5c93-4887-86bb-8e2b76c120bf\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " Nov 03 20:20:08.243776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.243715 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-credential-socket\") pod \"226bf833-ba6d-4c44-91bd-015817c60b68\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " Nov 03 20:20:08.243776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.243757 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-workload-certs\") pod \"30c48c95-5c93-4887-86bb-8e2b76c120bf\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " Nov 03 20:20:08.243890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.243840 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/226bf833-ba6d-4c44-91bd-015817c60b68-istio-token\" (UniqueName: \"kubernetes.io/projected/226bf833-ba6d-4c44-91bd-015817c60b68-istio-token\") pod \"226bf833-ba6d-4c44-91bd-015817c60b68\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " Nov 03 20:20:08.243949 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.243897 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/226bf833-ba6d-4c44-91bd-015817c60b68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226bf833-ba6d-4c44-91bd-015817c60b68-istiod-ca-cert\") pod \"226bf833-ba6d-4c44-91bd-015817c60b68\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " Nov 03 20:20:08.244009 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.243949 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c48c95-5c93-4887-86bb-8e2b76c120bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c48c95-5c93-4887-86bb-8e2b76c120bf-istiod-ca-cert\") pod \"30c48c95-5c93-4887-86bb-8e2b76c120bf\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " Nov 03 20:20:08.244009 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.243979 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-data\") pod \"30c48c95-5c93-4887-86bb-8e2b76c120bf\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " Nov 03 20:20:08.244130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.244036 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-workload-socket\") pod \"226bf833-ba6d-4c44-91bd-015817c60b68\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " Nov 03 20:20:08.244186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.244129 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-credential-socket\") pod \"30c48c95-5c93-4887-86bb-8e2b76c120bf\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " Nov 03 20:20:08.244246 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.244220 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/226bf833-ba6d-4c44-91bd-015817c60b68-kube-api-access-72rxl\" (UniqueName: \"kubernetes.io/projected/226bf833-ba6d-4c44-91bd-015817c60b68-kube-api-access-72rxl\") pod \"226bf833-ba6d-4c44-91bd-015817c60b68\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " Nov 03 20:20:08.244301 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.244280 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-workload-certs\") pod \"226bf833-ba6d-4c44-91bd-015817c60b68\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " Nov 03 20:20:08.244356 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.244343 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/226bf833-ba6d-4c44-91bd-015817c60b68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226bf833-ba6d-4c44-91bd-015817c60b68-istio-podinfo\") pod \"226bf833-ba6d-4c44-91bd-015817c60b68\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " Nov 03 20:20:08.244408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.244377 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-token\" (UniqueName: \"kubernetes.io/projected/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-token\") pod \"30c48c95-5c93-4887-86bb-8e2b76c120bf\" (UID: \"30c48c95-5c93-4887-86bb-8e2b76c120bf\") " Nov 03 20:20:08.244472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.244446 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-istio-envoy\") pod \"226bf833-ba6d-4c44-91bd-015817c60b68\" (UID: \"226bf833-ba6d-4c44-91bd-015817c60b68\") " Nov 03 20:20:08.244933 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.244884 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-workload-certs" pod "30c48c95-5c93-4887-86bb-8e2b76c120bf" (UID: "30c48c95-5c93-4887-86bb-8e2b76c120bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:08.246429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.245300 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226bf833-ba6d-4c44-91bd-015817c60b68-istiod-ca-cert" pod "226bf833-ba6d-4c44-91bd-015817c60b68" (UID: "226bf833-ba6d-4c44-91bd-015817c60b68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:08.246429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.245392 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.246429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.245413 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.246429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.245429 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.246429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.245601 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-data" pod "30c48c95-5c93-4887-86bb-8e2b76c120bf" (UID: "30c48c95-5c93-4887-86bb-8e2b76c120bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:08.246429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.246033 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c48c95-5c93-4887-86bb-8e2b76c120bf-istiod-ca-cert" pod "30c48c95-5c93-4887-86bb-8e2b76c120bf" (UID: "30c48c95-5c93-4887-86bb-8e2b76c120bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:08.246429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.246270 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-credential-socket" pod "226bf833-ba6d-4c44-91bd-015817c60b68" (UID: "226bf833-ba6d-4c44-91bd-015817c60b68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:08.247169 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.246636 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-workload-socket" pod "226bf833-ba6d-4c44-91bd-015817c60b68" (UID: "226bf833-ba6d-4c44-91bd-015817c60b68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:08.247311 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.247280 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-credential-socket" pod "30c48c95-5c93-4887-86bb-8e2b76c120bf" (UID: "30c48c95-5c93-4887-86bb-8e2b76c120bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:08.250195 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.249944 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-envoy" pod "30c48c95-5c93-4887-86bb-8e2b76c120bf" (UID: "30c48c95-5c93-4887-86bb-8e2b76c120bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:08.250488 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.250462 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-workload-certs" pod "226bf833-ba6d-4c44-91bd-015817c60b68" (UID: "226bf833-ba6d-4c44-91bd-015817c60b68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:08.250588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.250550 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-podinfo" pod "30c48c95-5c93-4887-86bb-8e2b76c120bf" (UID: "30c48c95-5c93-4887-86bb-8e2b76c120bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:20:08.251361 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.251325 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/226bf833-ba6d-4c44-91bd-015817c60b68-istio-podinfo" pod "226bf833-ba6d-4c44-91bd-015817c60b68" (UID: "226bf833-ba6d-4c44-91bd-015817c60b68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:20:08.251744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.251701 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c48c95-5c93-4887-86bb-8e2b76c120bf-kube-api-access-8x7cl" pod "30c48c95-5c93-4887-86bb-8e2b76c120bf" (UID: "30c48c95-5c93-4887-86bb-8e2b76c120bf"). InnerVolumeSpecName "kube-api-access-8x7cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:08.262970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.255759 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-istio-envoy" pod "226bf833-ba6d-4c44-91bd-015817c60b68" (UID: "226bf833-ba6d-4c44-91bd-015817c60b68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:08.263736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.263696 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-token" pod "30c48c95-5c93-4887-86bb-8e2b76c120bf" (UID: "30c48c95-5c93-4887-86bb-8e2b76c120bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:08.268776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.268724 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226bf833-ba6d-4c44-91bd-015817c60b68-kube-api-access-72rxl" pod "226bf833-ba6d-4c44-91bd-015817c60b68" (UID: "226bf833-ba6d-4c44-91bd-015817c60b68"). InnerVolumeSpecName "kube-api-access-72rxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:08.270597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.270509 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226bf833-ba6d-4c44-91bd-015817c60b68-istio-token" pod "226bf833-ba6d-4c44-91bd-015817c60b68" (UID: "226bf833-ba6d-4c44-91bd-015817c60b68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:08.344495 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.344461 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" Nov 03 20:20:08.346047 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.346015 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-token\" (UniqueName: \"kubernetes.io/projected/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-token\") pod \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " Nov 03 20:20:08.346206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.346135 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-podinfo\") pod \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " Nov 03 20:20:08.346270 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.346204 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-kube-api-access-glrmz\" (UniqueName: \"kubernetes.io/projected/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-kube-api-access-glrmz\") pod \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " Nov 03 20:20:08.346340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.346325 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-workload-certs\") pod \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " Nov 03 20:20:08.346425 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.346411 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-credential-socket\") pod \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " Nov 03 20:20:08.346483 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.346459 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-envoy\") pod \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " Nov 03 20:20:08.347029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.347006 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istiod-ca-cert\") pod \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " Nov 03 20:20:08.347158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.347055 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-data\") pod \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " Nov 03 20:20:08.347227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.347171 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-workload-socket\") pod \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\" (UID: \"c58995fe-f4f0-48be-b3a0-03afb09e2ad3\") " Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348163 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72rxl\" (UniqueName: \"kubernetes.io/projected/226bf833-ba6d-4c44-91bd-015817c60b68-kube-api-access-72rxl\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348192 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348208 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226bf833-ba6d-4c44-91bd-015817c60b68-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348223 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348237 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348252 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348267 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348281 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x7cl\" (UniqueName: \"kubernetes.io/projected/30c48c95-5c93-4887-86bb-8e2b76c120bf-kube-api-access-8x7cl\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348296 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348309 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/226bf833-ba6d-4c44-91bd-015817c60b68-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348323 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226bf833-ba6d-4c44-91bd-015817c60b68-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348337 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c48c95-5c93-4887-86bb-8e2b76c120bf-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348352 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348366 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226bf833-ba6d-4c44-91bd-015817c60b68-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348381 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c48c95-5c93-4887-86bb-8e2b76c120bf-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348699 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-workload-socket" pod "c58995fe-f4f0-48be-b3a0-03afb09e2ad3" (UID: "c58995fe-f4f0-48be-b3a0-03afb09e2ad3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.348897 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-workload-certs" pod "c58995fe-f4f0-48be-b3a0-03afb09e2ad3" (UID: "c58995fe-f4f0-48be-b3a0-03afb09e2ad3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.349125 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-credential-socket" pod "c58995fe-f4f0-48be-b3a0-03afb09e2ad3" (UID: "c58995fe-f4f0-48be-b3a0-03afb09e2ad3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.349450 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-kube-api-access-glrmz" pod "c58995fe-f4f0-48be-b3a0-03afb09e2ad3" (UID: "c58995fe-f4f0-48be-b3a0-03afb09e2ad3"). InnerVolumeSpecName "kube-api-access-glrmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.350017 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istiod-ca-cert" pod "c58995fe-f4f0-48be-b3a0-03afb09e2ad3" (UID: "c58995fe-f4f0-48be-b3a0-03afb09e2ad3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:08.354248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.350080 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-data" pod "c58995fe-f4f0-48be-b3a0-03afb09e2ad3" (UID: "c58995fe-f4f0-48be-b3a0-03afb09e2ad3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:08.373924 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.373824 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-token" pod "c58995fe-f4f0-48be-b3a0-03afb09e2ad3" (UID: "c58995fe-f4f0-48be-b3a0-03afb09e2ad3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:08.374072 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.373978 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-envoy" pod "c58995fe-f4f0-48be-b3a0-03afb09e2ad3" (UID: "c58995fe-f4f0-48be-b3a0-03afb09e2ad3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:08.375781 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.375737 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-podinfo" pod "c58995fe-f4f0-48be-b3a0-03afb09e2ad3" (UID: "c58995fe-f4f0-48be-b3a0-03afb09e2ad3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:20:08.449712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.449675 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ef63c4d-f39d-4623-869c-26983ba16e10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef63c4d-f39d-4623-869c-26983ba16e10-maas-api-xks-ca\") pod \"7ef63c4d-f39d-4623-869c-26983ba16e10\" (UID: \"7ef63c4d-f39d-4623-869c-26983ba16e10\") " Nov 03 20:20:08.449889 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.449768 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ef63c4d-f39d-4623-869c-26983ba16e10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef63c4d-f39d-4623-869c-26983ba16e10-maas-api-openshift-ca\") pod \"7ef63c4d-f39d-4623-869c-26983ba16e10\" (UID: \"7ef63c4d-f39d-4623-869c-26983ba16e10\") " Nov 03 20:20:08.449947 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.449929 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef63c4d-f39d-4623-869c-26983ba16e10-kube-api-access-9csx2\" (UniqueName: \"kubernetes.io/projected/7ef63c4d-f39d-4623-869c-26983ba16e10-kube-api-access-9csx2\") pod \"7ef63c4d-f39d-4623-869c-26983ba16e10\" (UID: \"7ef63c4d-f39d-4623-869c-26983ba16e10\") " Nov 03 20:20:08.450720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.450691 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef63c4d-f39d-4623-869c-26983ba16e10-maas-api-openshift-ca" pod "7ef63c4d-f39d-4623-869c-26983ba16e10" (UID: "7ef63c4d-f39d-4623-869c-26983ba16e10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:08.452188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.451052 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.452188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.451348 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.452188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.451955 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.452188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.451984 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.452188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.451996 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.452188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.452008 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glrmz\" (UniqueName: \"kubernetes.io/projected/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-kube-api-access-glrmz\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.452188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.452022 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.452188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.452036 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.452188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.452050 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58995fe-f4f0-48be-b3a0-03afb09e2ad3-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.453347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.453308 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ef63c4d-f39d-4623-869c-26983ba16e10-maas-api-xks-ca" pod "7ef63c4d-f39d-4623-869c-26983ba16e10" (UID: "7ef63c4d-f39d-4623-869c-26983ba16e10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:08.453693 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.453665 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef63c4d-f39d-4623-869c-26983ba16e10-kube-api-access-9csx2" pod "7ef63c4d-f39d-4623-869c-26983ba16e10" (UID: "7ef63c4d-f39d-4623-869c-26983ba16e10"). InnerVolumeSpecName "kube-api-access-9csx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:08.554419 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.554317 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9csx2\" (UniqueName: \"kubernetes.io/projected/7ef63c4d-f39d-4623-869c-26983ba16e10-kube-api-access-9csx2\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.554419 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.554371 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ef63c4d-f39d-4623-869c-26983ba16e10-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:08.554419 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:08.554393 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ef63c4d-f39d-4623-869c-26983ba16e10-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:09.016691 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.016622 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:20:09.017922 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.017616 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" podUID="30c48c95-5c93-4887-86bb-8e2b76c120bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:20:09.026597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.020670 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" event={"ID":"226bf833-ba6d-4c44-91bd-015817c60b68","Type":"ContainerDied","Data":"c1fc201a572eb5617d3528d20b5643fb3e469ddeda8104b01205a9516000632b"} Nov 03 20:20:09.026597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.020810 5111 scope.go:122] "RemoveContainer" containerID="f6248dfd3770b44252cc97107410e9e9182c70851aaaaba6871bb8810d841edb" Nov 03 20:20:09.026597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.021048 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" Nov 03 20:20:09.095138 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.089432 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" event={"ID":"7ef63c4d-f39d-4623-869c-26983ba16e10","Type":"ContainerDied","Data":"f8777905b3301790491c2acb9f3bb636c93756cc926bbafab75d6fb3b998e9a8"} Nov 03 20:20:09.095138 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.089509 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8777905b3301790491c2acb9f3bb636c93756cc926bbafab75d6fb3b998e9a8" Nov 03 20:20:09.095138 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.089638 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js" Nov 03 20:20:09.099668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.099634 5111 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:20:09.100751 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.100721 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j" podUID="226bf833-ba6d-4c44-91bd-015817c60b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:20:09.100960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.100240 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" Nov 03 20:20:09.107273 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.107251 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" Nov 03 20:20:09.118870 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.100139 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f" event={"ID":"30c48c95-5c93-4887-86bb-8e2b76c120bf","Type":"ContainerDied","Data":"281d4a37f07b025f0e770cdd92fedd7d98fe780fcfad967340d419454b9dcfa2"} Nov 03 20:20:09.119043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.118909 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc" event={"ID":"c58995fe-f4f0-48be-b3a0-03afb09e2ad3","Type":"ContainerDied","Data":"14ad73a5c0272adf7cfd4e1148418af797f58b6f7e7b3ec6aca14bd9ae6ae4d9"} Nov 03 20:20:09.190596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.186844 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f"] Nov 03 20:20:09.190596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.187503 5111 scope.go:122] "RemoveContainer" containerID="958d342d9e3d0f5b764ac4389a7c94be62f916c4d6365d2c92ae7838f972db26" Nov 03 20:20:09.197579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.196303 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-95x7f"] Nov 03 20:20:09.206674 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.206651 5111 scope.go:122] "RemoveContainer" containerID="00327e7941e5036b29f28150fa92dd0818abad17290fe7f20147fe7926729a0c" Nov 03 20:20:09.208093 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.208076 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j"] Nov 03 20:20:09.215346 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.215308 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-djv8j"] Nov 03 20:20:09.223453 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.223430 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc"] Nov 03 20:20:09.229766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.229742 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-wk6gc"] Nov 03 20:20:09.269479 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.269397 5111 patch_prober.go:29] interesting pod/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 20:20:09.269479 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.269467 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" podUID="a3e211eb-7ac1-4136-b8d0-65211bb869f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 20:20:09.397117 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.396715 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd"] Nov 03 20:20:09.406796 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.406716 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz"] Nov 03 20:20:09.407887 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.407860 5111 cpu_manager.go:414] "containerMap: removing container" podUID="7ef63c4d-f39d-4623-869c-26983ba16e10" containerName="revoke-keys" Nov 03 20:20:09.407887 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.407885 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef63c4d-f39d-4623-869c-26983ba16e10" containerName="revoke-keys" Nov 03 20:20:09.408045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.407907 5111 cpu_manager.go:414] "containerMap: removing container" podUID="30c48c95-5c93-4887-86bb-8e2b76c120bf" containerName="istio-proxy" Nov 03 20:20:09.408045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.407913 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c48c95-5c93-4887-86bb-8e2b76c120bf" containerName="istio-proxy" Nov 03 20:20:09.408045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.407926 5111 cpu_manager.go:414] "containerMap: removing container" podUID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" containerName="istio-proxy" Nov 03 20:20:09.408045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.407931 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" containerName="istio-proxy" Nov 03 20:20:09.408045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.407969 5111 cpu_manager.go:414] "containerMap: removing container" podUID="226bf833-ba6d-4c44-91bd-015817c60b68" containerName="istio-proxy" Nov 03 20:20:09.408045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.407976 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="226bf833-ba6d-4c44-91bd-015817c60b68" containerName="istio-proxy" Nov 03 20:20:09.408045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.408002 5111 cpu_manager.go:414] "containerMap: removing container" podUID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" containerName="istio-proxy" Nov 03 20:20:09.408045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.408006 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" containerName="istio-proxy" Nov 03 20:20:09.408293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.408122 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c48c95-5c93-4887-86bb-8e2b76c120bf" containerName="istio-proxy" Nov 03 20:20:09.408293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.408130 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" containerName="istio-proxy" Nov 03 20:20:09.408293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.408142 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="226bf833-ba6d-4c44-91bd-015817c60b68" containerName="istio-proxy" Nov 03 20:20:09.408293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.408152 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef63c4d-f39d-4623-869c-26983ba16e10" containerName="revoke-keys" Nov 03 20:20:09.412801 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.412757 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz" Nov 03 20:20:09.436132 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.436105 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz"] Nov 03 20:20:09.452519 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.452495 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:20:09.452668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.452636 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:20:09.531911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.531816 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecc21eef-fb5a-4543-a596-12352ea7b48c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz\" (UID: \"ecc21eef-fb5a-4543-a596-12352ea7b48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz" Nov 03 20:20:09.532200 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.532175 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecc21eef-fb5a-4543-a596-12352ea7b48c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz\" (UID: \"ecc21eef-fb5a-4543-a596-12352ea7b48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz" Nov 03 20:20:09.532332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.532233 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5n7w\" (UniqueName: \"kubernetes.io/projected/ecc21eef-fb5a-4543-a596-12352ea7b48c-kube-api-access-d5n7w\") pod \"maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz\" (UID: \"ecc21eef-fb5a-4543-a596-12352ea7b48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz" Nov 03 20:20:09.640751 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.634501 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecc21eef-fb5a-4543-a596-12352ea7b48c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz\" (UID: \"ecc21eef-fb5a-4543-a596-12352ea7b48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz" Nov 03 20:20:09.641288 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.641234 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5n7w\" (UniqueName: \"kubernetes.io/projected/ecc21eef-fb5a-4543-a596-12352ea7b48c-kube-api-access-d5n7w\") pod \"maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz\" (UID: \"ecc21eef-fb5a-4543-a596-12352ea7b48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz" Nov 03 20:20:09.641653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.641630 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecc21eef-fb5a-4543-a596-12352ea7b48c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz\" (UID: \"ecc21eef-fb5a-4543-a596-12352ea7b48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz" Nov 03 20:20:09.644844 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.644803 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecc21eef-fb5a-4543-a596-12352ea7b48c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz\" (UID: \"ecc21eef-fb5a-4543-a596-12352ea7b48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz" Nov 03 20:20:09.647978 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.647944 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecc21eef-fb5a-4543-a596-12352ea7b48c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz\" (UID: \"ecc21eef-fb5a-4543-a596-12352ea7b48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz" Nov 03 20:20:09.652306 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.652280 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5n7w\" (UniqueName: \"kubernetes.io/projected/ecc21eef-fb5a-4543-a596-12352ea7b48c-kube-api-access-d5n7w\") pod \"maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz\" (UID: \"ecc21eef-fb5a-4543-a596-12352ea7b48c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz" Nov 03 20:20:09.681924 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.681893 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:20:09.741345 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.741305 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz" Nov 03 20:20:09.845676 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.845637 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-podinfo\") pod \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " Nov 03 20:20:09.845849 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.845705 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-workload-certs\") pod \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " Nov 03 20:20:09.845849 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.845765 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istiod-ca-cert\") pod \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " Nov 03 20:20:09.845849 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.845793 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-credential-socket\") pod \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " Nov 03 20:20:09.845849 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.845829 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-envoy\") pod \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " Nov 03 20:20:09.846120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.845897 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-token\" (UniqueName: \"kubernetes.io/projected/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-token\") pod \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " Nov 03 20:20:09.846120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.845969 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-data\") pod \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " Nov 03 20:20:09.846120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.846113 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e211eb-7ac1-4136-b8d0-65211bb869f6-kube-api-access-2qf6j\" (UniqueName: \"kubernetes.io/projected/a3e211eb-7ac1-4136-b8d0-65211bb869f6-kube-api-access-2qf6j\") pod \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " Nov 03 20:20:09.846290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.846167 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-workload-socket\") pod \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\" (UID: \"a3e211eb-7ac1-4136-b8d0-65211bb869f6\") " Nov 03 20:20:09.846478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.846449 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-credential-socket" pod "a3e211eb-7ac1-4136-b8d0-65211bb869f6" (UID: "a3e211eb-7ac1-4136-b8d0-65211bb869f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:09.846940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.846674 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-workload-socket" pod "a3e211eb-7ac1-4136-b8d0-65211bb869f6" (UID: "a3e211eb-7ac1-4136-b8d0-65211bb869f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:09.846940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.846794 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-data" pod "a3e211eb-7ac1-4136-b8d0-65211bb869f6" (UID: "a3e211eb-7ac1-4136-b8d0-65211bb869f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:09.846940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.846825 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istiod-ca-cert" pod "a3e211eb-7ac1-4136-b8d0-65211bb869f6" (UID: "a3e211eb-7ac1-4136-b8d0-65211bb869f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:09.847268 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.847155 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-workload-certs" pod "a3e211eb-7ac1-4136-b8d0-65211bb869f6" (UID: "a3e211eb-7ac1-4136-b8d0-65211bb869f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:09.847684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.847658 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:09.847787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.847688 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:09.847787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.847704 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:09.847787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.847719 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:09.847787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.847733 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:09.849728 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.849700 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-podinfo" pod "a3e211eb-7ac1-4136-b8d0-65211bb869f6" (UID: "a3e211eb-7ac1-4136-b8d0-65211bb869f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:20:09.849882 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.849861 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-token" pod "a3e211eb-7ac1-4136-b8d0-65211bb869f6" (UID: "a3e211eb-7ac1-4136-b8d0-65211bb869f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:09.850482 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.850453 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-envoy" pod "a3e211eb-7ac1-4136-b8d0-65211bb869f6" (UID: "a3e211eb-7ac1-4136-b8d0-65211bb869f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:09.850686 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.850664 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e211eb-7ac1-4136-b8d0-65211bb869f6-kube-api-access-2qf6j" pod "a3e211eb-7ac1-4136-b8d0-65211bb869f6" (UID: "a3e211eb-7ac1-4136-b8d0-65211bb869f6"). InnerVolumeSpecName "kube-api-access-2qf6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:09.950374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.950336 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qf6j\" (UniqueName: \"kubernetes.io/projected/a3e211eb-7ac1-4136-b8d0-65211bb869f6-kube-api-access-2qf6j\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:09.950374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.950375 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:09.950707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.950391 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:09.950707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:09.950405 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3e211eb-7ac1-4136-b8d0-65211bb869f6-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:10.037511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:10.037474 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz"] Nov 03 20:20:10.043022 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:20:10.042991 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc21eef_fb5a_4543_a596_12352ea7b48c.slice/crio-a1785b70bfd054a631c7ac18231a8a0b6a78da659621d6cad1f11524ca675845 WatchSource:0}: Error finding container a1785b70bfd054a631c7ac18231a8a0b6a78da659621d6cad1f11524ca675845: Status 404 returned error can't find the container with id a1785b70bfd054a631c7ac18231a8a0b6a78da659621d6cad1f11524ca675845 Nov 03 20:20:10.117714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:10.117682 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz" event={"ID":"ecc21eef-fb5a-4543-a596-12352ea7b48c","Type":"ContainerStarted","Data":"a1785b70bfd054a631c7ac18231a8a0b6a78da659621d6cad1f11524ca675845"} Nov 03 20:20:10.121451 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:10.121420 5111 generic.go:356] "Generic (PLEG): container finished" podID="a3e211eb-7ac1-4136-b8d0-65211bb869f6" containerID="846b3c01b9cb7f1dac233ae831f1943fad718283e588e39e894e632fec55f557" exitCode=0 Nov 03 20:20:10.121680 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:10.121466 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" Nov 03 20:20:10.121680 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:10.121501 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" event={"ID":"a3e211eb-7ac1-4136-b8d0-65211bb869f6","Type":"ContainerDied","Data":"846b3c01b9cb7f1dac233ae831f1943fad718283e588e39e894e632fec55f557"} Nov 03 20:20:10.121680 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:10.121568 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck" event={"ID":"a3e211eb-7ac1-4136-b8d0-65211bb869f6","Type":"ContainerDied","Data":"e3613d33dfca140eb0d426c1e636cc5e115db5f9b87377d09799db2996038e7d"} Nov 03 20:20:10.121680 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:10.121583 5111 scope.go:122] "RemoveContainer" containerID="846b3c01b9cb7f1dac233ae831f1943fad718283e588e39e894e632fec55f557" Nov 03 20:20:10.134614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:10.134587 5111 scope.go:122] "RemoveContainer" containerID="846b3c01b9cb7f1dac233ae831f1943fad718283e588e39e894e632fec55f557" Nov 03 20:20:10.134943 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:20:10.134923 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846b3c01b9cb7f1dac233ae831f1943fad718283e588e39e894e632fec55f557\": container with ID starting with 846b3c01b9cb7f1dac233ae831f1943fad718283e588e39e894e632fec55f557 not found: ID does not exist" containerID="846b3c01b9cb7f1dac233ae831f1943fad718283e588e39e894e632fec55f557" Nov 03 20:20:10.135000 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:10.134953 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846b3c01b9cb7f1dac233ae831f1943fad718283e588e39e894e632fec55f557"} err="failed to get container status \"846b3c01b9cb7f1dac233ae831f1943fad718283e588e39e894e632fec55f557\": rpc error: code = NotFound desc = could not find container \"846b3c01b9cb7f1dac233ae831f1943fad718283e588e39e894e632fec55f557\": container with ID starting with 846b3c01b9cb7f1dac233ae831f1943fad718283e588e39e894e632fec55f557 not found: ID does not exist" Nov 03 20:20:10.150127 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:10.150094 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck"] Nov 03 20:20:10.154488 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:10.154465 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-98tck"] Nov 03 20:20:10.188191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:10.188155 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="226bf833-ba6d-4c44-91bd-015817c60b68" path="/var/lib/kubelet/pods/226bf833-ba6d-4c44-91bd-015817c60b68/volumes" Nov 03 20:20:10.189049 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:10.189028 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c48c95-5c93-4887-86bb-8e2b76c120bf" path="/var/lib/kubelet/pods/30c48c95-5c93-4887-86bb-8e2b76c120bf/volumes" Nov 03 20:20:10.189776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:10.189754 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3e211eb-7ac1-4136-b8d0-65211bb869f6" path="/var/lib/kubelet/pods/a3e211eb-7ac1-4136-b8d0-65211bb869f6/volumes" Nov 03 20:20:10.190964 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:10.190940 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" path="/var/lib/kubelet/pods/c58995fe-f4f0-48be-b3a0-03afb09e2ad3/volumes" Nov 03 20:20:11.132398 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:11.132365 5111 generic.go:356] "Generic (PLEG): container finished" podID="ecc21eef-fb5a-4543-a596-12352ea7b48c" containerID="b8b6149a8b76c68a93c283d2a15b48ed4cedf8520cf0d2fd8c9dd5dedf75cf66" exitCode=0 Nov 03 20:20:11.132860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:11.132445 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz" event={"ID":"ecc21eef-fb5a-4543-a596-12352ea7b48c","Type":"ContainerDied","Data":"b8b6149a8b76c68a93c283d2a15b48ed4cedf8520cf0d2fd8c9dd5dedf75cf66"} Nov 03 20:20:11.416750 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:11.416711 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p"] Nov 03 20:20:11.420694 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:11.420670 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-663013-6w92p"] Nov 03 20:20:12.187332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:12.187295 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d5b785-00c4-40ec-b265-af208e4cc152" path="/var/lib/kubelet/pods/69d5b785-00c4-40ec-b265-af208e4cc152/volumes" Nov 03 20:20:12.358293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:12.358266 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz" Nov 03 20:20:12.408613 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:12.408518 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc21eef-fb5a-4543-a596-12352ea7b48c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecc21eef-fb5a-4543-a596-12352ea7b48c-maas-api-openshift-ca\") pod \"ecc21eef-fb5a-4543-a596-12352ea7b48c\" (UID: \"ecc21eef-fb5a-4543-a596-12352ea7b48c\") " Nov 03 20:20:12.408613 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:12.408579 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc21eef-fb5a-4543-a596-12352ea7b48c-kube-api-access-d5n7w\" (UniqueName: \"kubernetes.io/projected/ecc21eef-fb5a-4543-a596-12352ea7b48c-kube-api-access-d5n7w\") pod \"ecc21eef-fb5a-4543-a596-12352ea7b48c\" (UID: \"ecc21eef-fb5a-4543-a596-12352ea7b48c\") " Nov 03 20:20:12.408824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:12.408618 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecc21eef-fb5a-4543-a596-12352ea7b48c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecc21eef-fb5a-4543-a596-12352ea7b48c-maas-api-xks-ca\") pod \"ecc21eef-fb5a-4543-a596-12352ea7b48c\" (UID: \"ecc21eef-fb5a-4543-a596-12352ea7b48c\") " Nov 03 20:20:12.408979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:12.408950 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc21eef-fb5a-4543-a596-12352ea7b48c-maas-api-openshift-ca" pod "ecc21eef-fb5a-4543-a596-12352ea7b48c" (UID: "ecc21eef-fb5a-4543-a596-12352ea7b48c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:12.409483 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:12.409459 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecc21eef-fb5a-4543-a596-12352ea7b48c-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:12.410770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:12.410746 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecc21eef-fb5a-4543-a596-12352ea7b48c-maas-api-xks-ca" pod "ecc21eef-fb5a-4543-a596-12352ea7b48c" (UID: "ecc21eef-fb5a-4543-a596-12352ea7b48c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:12.410920 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:12.410897 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc21eef-fb5a-4543-a596-12352ea7b48c-kube-api-access-d5n7w" pod "ecc21eef-fb5a-4543-a596-12352ea7b48c" (UID: "ecc21eef-fb5a-4543-a596-12352ea7b48c"). InnerVolumeSpecName "kube-api-access-d5n7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:12.511221 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:12.511180 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5n7w\" (UniqueName: \"kubernetes.io/projected/ecc21eef-fb5a-4543-a596-12352ea7b48c-kube-api-access-d5n7w\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:12.511221 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:12.511221 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecc21eef-fb5a-4543-a596-12352ea7b48c-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:13.154600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:13.154564 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz" Nov 03 20:20:13.154821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:13.154596 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz" event={"ID":"ecc21eef-fb5a-4543-a596-12352ea7b48c","Type":"ContainerDied","Data":"a1785b70bfd054a631c7ac18231a8a0b6a78da659621d6cad1f11524ca675845"} Nov 03 20:20:13.154821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:13.154664 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1785b70bfd054a631c7ac18231a8a0b6a78da659621d6cad1f11524ca675845" Nov 03 20:20:14.833822 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:14.833783 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx"] Nov 03 20:20:16.635030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.634994 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d"] Nov 03 20:20:16.635765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.635749 5111 cpu_manager.go:414] "containerMap: removing container" podUID="a3e211eb-7ac1-4136-b8d0-65211bb869f6" containerName="istio-proxy" Nov 03 20:20:16.635765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.635765 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e211eb-7ac1-4136-b8d0-65211bb869f6" containerName="istio-proxy" Nov 03 20:20:16.635907 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.635793 5111 cpu_manager.go:414] "containerMap: removing container" podUID="ecc21eef-fb5a-4543-a596-12352ea7b48c" containerName="revoke-keys" Nov 03 20:20:16.635907 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.635798 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc21eef-fb5a-4543-a596-12352ea7b48c" containerName="revoke-keys" Nov 03 20:20:16.635984 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.635974 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e211eb-7ac1-4136-b8d0-65211bb869f6" containerName="istio-proxy" Nov 03 20:20:16.637169 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.635989 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58995fe-f4f0-48be-b3a0-03afb09e2ad3" containerName="istio-proxy" Nov 03 20:20:16.637169 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.635997 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc21eef-fb5a-4543-a596-12352ea7b48c" containerName="revoke-keys" Nov 03 20:20:16.640345 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.640324 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" Nov 03 20:20:16.648729 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.648706 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d"] Nov 03 20:20:16.649120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.649100 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:20:16.649316 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.649296 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:20:16.653929 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.653902 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj"] Nov 03 20:20:16.665623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.665595 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" Nov 03 20:20:16.668805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.668780 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj"] Nov 03 20:20:16.707775 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.707733 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85998d26-9a30-423e-91c8-0926e2f724ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e920ee1-bmf6d\" (UID: \"85998d26-9a30-423e-91c8-0926e2f724ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" Nov 03 20:20:16.707954 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.707784 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85998d26-9a30-423e-91c8-0926e2f724ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e920ee1-bmf6d\" (UID: \"85998d26-9a30-423e-91c8-0926e2f724ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" Nov 03 20:20:16.708020 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.707947 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc9sv\" (UniqueName: \"kubernetes.io/projected/85998d26-9a30-423e-91c8-0926e2f724ae-kube-api-access-tc9sv\") pod \"maas-api-revoke-sub-6e920ee1-bmf6d\" (UID: \"85998d26-9a30-423e-91c8-0926e2f724ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" Nov 03 20:20:16.809631 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.809583 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49rk7\" (UniqueName: \"kubernetes.io/projected/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-kube-api-access-49rk7\") pod \"maas-api-revoke-sub-427da27c-dkffj\" (UID: \"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" Nov 03 20:20:16.809875 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.809850 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-427da27c-dkffj\" (UID: \"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" Nov 03 20:20:16.809945 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.809888 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-427da27c-dkffj\" (UID: \"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" Nov 03 20:20:16.810006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.809968 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85998d26-9a30-423e-91c8-0926e2f724ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e920ee1-bmf6d\" (UID: \"85998d26-9a30-423e-91c8-0926e2f724ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" Nov 03 20:20:16.810006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.809987 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85998d26-9a30-423e-91c8-0926e2f724ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e920ee1-bmf6d\" (UID: \"85998d26-9a30-423e-91c8-0926e2f724ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" Nov 03 20:20:16.810157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.810032 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc9sv\" (UniqueName: \"kubernetes.io/projected/85998d26-9a30-423e-91c8-0926e2f724ae-kube-api-access-tc9sv\") pod \"maas-api-revoke-sub-6e920ee1-bmf6d\" (UID: \"85998d26-9a30-423e-91c8-0926e2f724ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" Nov 03 20:20:16.810808 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.810782 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85998d26-9a30-423e-91c8-0926e2f724ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e920ee1-bmf6d\" (UID: \"85998d26-9a30-423e-91c8-0926e2f724ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" Nov 03 20:20:16.812616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.812595 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85998d26-9a30-423e-91c8-0926e2f724ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e920ee1-bmf6d\" (UID: \"85998d26-9a30-423e-91c8-0926e2f724ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" Nov 03 20:20:16.818516 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.818494 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc9sv\" (UniqueName: \"kubernetes.io/projected/85998d26-9a30-423e-91c8-0926e2f724ae-kube-api-access-tc9sv\") pod \"maas-api-revoke-sub-6e920ee1-bmf6d\" (UID: \"85998d26-9a30-423e-91c8-0926e2f724ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" Nov 03 20:20:16.911264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.911188 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-427da27c-dkffj\" (UID: \"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" Nov 03 20:20:16.911264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.911229 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-427da27c-dkffj\" (UID: \"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" Nov 03 20:20:16.911442 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.911393 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49rk7\" (UniqueName: \"kubernetes.io/projected/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-kube-api-access-49rk7\") pod \"maas-api-revoke-sub-427da27c-dkffj\" (UID: \"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" Nov 03 20:20:16.911898 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.911878 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-427da27c-dkffj\" (UID: \"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" Nov 03 20:20:16.913794 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.913772 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-427da27c-dkffj\" (UID: \"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" Nov 03 20:20:16.919358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.919337 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49rk7\" (UniqueName: \"kubernetes.io/projected/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-kube-api-access-49rk7\") pod \"maas-api-revoke-sub-427da27c-dkffj\" (UID: \"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" Nov 03 20:20:16.962939 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.962914 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" Nov 03 20:20:16.987006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:16.986980 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" Nov 03 20:20:17.296316 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:20:17.296287 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85998d26_9a30_423e_91c8_0926e2f724ae.slice/crio-4b623af387f0add41ac322d02fb4a23f0c5ca75b0618a04bb7e3a4cdd6d8df54 WatchSource:0}: Error finding container 4b623af387f0add41ac322d02fb4a23f0c5ca75b0618a04bb7e3a4cdd6d8df54: Status 404 returned error can't find the container with id 4b623af387f0add41ac322d02fb4a23f0c5ca75b0618a04bb7e3a4cdd6d8df54 Nov 03 20:20:17.296316 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:17.296302 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d"] Nov 03 20:20:17.354393 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:17.354366 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj"] Nov 03 20:20:17.447612 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:20:17.447524 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b5ec9f_f8a2_4b80_b1ca_e405a46aa8ad.slice/crio-8f8b7f3c684a306a3f5201a8b55246dd21645881b9d9bbd04fccbda59794f753 WatchSource:0}: Error finding container 8f8b7f3c684a306a3f5201a8b55246dd21645881b9d9bbd04fccbda59794f753: Status 404 returned error can't find the container with id 8f8b7f3c684a306a3f5201a8b55246dd21645881b9d9bbd04fccbda59794f753 Nov 03 20:20:18.265449 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:18.265413 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" event={"ID":"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad","Type":"ContainerStarted","Data":"5564cf8e282cd1db6698f4762e45bdc64f8d3b755235ea513403e31ad68ecb1b"} Nov 03 20:20:18.265893 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:18.265476 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" event={"ID":"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad","Type":"ContainerStarted","Data":"8f8b7f3c684a306a3f5201a8b55246dd21645881b9d9bbd04fccbda59794f753"} Nov 03 20:20:18.267274 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:18.267247 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" event={"ID":"85998d26-9a30-423e-91c8-0926e2f724ae","Type":"ContainerStarted","Data":"4864f0938dd6a7308d3f8d9601e79c36dba0d3ce9f8f151229aa6e5410381281"} Nov 03 20:20:18.267383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:18.267297 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" event={"ID":"85998d26-9a30-423e-91c8-0926e2f724ae","Type":"ContainerStarted","Data":"4b623af387f0add41ac322d02fb4a23f0c5ca75b0618a04bb7e3a4cdd6d8df54"} Nov 03 20:20:18.285423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:18.285375 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" podStartSLOduration=2.285363375 podStartE2EDuration="2.285363375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:18.282839594 +0000 UTC m=+2448.245218034" watchObservedRunningTime="2026-11-03 20:20:18.285363375 +0000 UTC m=+2448.247741812" Nov 03 20:20:19.277693 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.277654 5111 generic.go:356] "Generic (PLEG): container finished" podID="85998d26-9a30-423e-91c8-0926e2f724ae" containerID="4864f0938dd6a7308d3f8d9601e79c36dba0d3ce9f8f151229aa6e5410381281" exitCode=0 Nov 03 20:20:19.278158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.277737 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" event={"ID":"85998d26-9a30-423e-91c8-0926e2f724ae","Type":"ContainerDied","Data":"4864f0938dd6a7308d3f8d9601e79c36dba0d3ce9f8f151229aa6e5410381281"} Nov 03 20:20:19.279768 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.279736 5111 generic.go:356] "Generic (PLEG): container finished" podID="97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad" containerID="5564cf8e282cd1db6698f4762e45bdc64f8d3b755235ea513403e31ad68ecb1b" exitCode=0 Nov 03 20:20:19.279886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.279780 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" event={"ID":"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad","Type":"ContainerDied","Data":"5564cf8e282cd1db6698f4762e45bdc64f8d3b755235ea513403e31ad68ecb1b"} Nov 03 20:20:19.375393 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.375354 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z"] Nov 03 20:20:19.394996 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.394906 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z"] Nov 03 20:20:19.394996 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.394974 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z" Nov 03 20:20:19.402958 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.402928 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2x8n\" (UniqueName: \"kubernetes.io/projected/30d2755a-e68d-4924-bbdd-6b0bd29d6398-kube-api-access-w2x8n\") pod \"maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z\" (UID: \"30d2755a-e68d-4924-bbdd-6b0bd29d6398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z" Nov 03 20:20:19.403370 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.403351 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d2755a-e68d-4924-bbdd-6b0bd29d6398-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z\" (UID: \"30d2755a-e68d-4924-bbdd-6b0bd29d6398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z" Nov 03 20:20:19.403557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.403526 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d2755a-e68d-4924-bbdd-6b0bd29d6398-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z\" (UID: \"30d2755a-e68d-4924-bbdd-6b0bd29d6398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z" Nov 03 20:20:19.505746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.505713 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2x8n\" (UniqueName: \"kubernetes.io/projected/30d2755a-e68d-4924-bbdd-6b0bd29d6398-kube-api-access-w2x8n\") pod \"maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z\" (UID: \"30d2755a-e68d-4924-bbdd-6b0bd29d6398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z" Nov 03 20:20:19.505914 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.505830 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d2755a-e68d-4924-bbdd-6b0bd29d6398-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z\" (UID: \"30d2755a-e68d-4924-bbdd-6b0bd29d6398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z" Nov 03 20:20:19.505914 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.505882 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d2755a-e68d-4924-bbdd-6b0bd29d6398-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z\" (UID: \"30d2755a-e68d-4924-bbdd-6b0bd29d6398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z" Nov 03 20:20:19.506826 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.506802 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d2755a-e68d-4924-bbdd-6b0bd29d6398-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z\" (UID: \"30d2755a-e68d-4924-bbdd-6b0bd29d6398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z" Nov 03 20:20:19.508591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.508568 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d2755a-e68d-4924-bbdd-6b0bd29d6398-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z\" (UID: \"30d2755a-e68d-4924-bbdd-6b0bd29d6398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z" Nov 03 20:20:19.513355 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.513335 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2x8n\" (UniqueName: \"kubernetes.io/projected/30d2755a-e68d-4924-bbdd-6b0bd29d6398-kube-api-access-w2x8n\") pod \"maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z\" (UID: \"30d2755a-e68d-4924-bbdd-6b0bd29d6398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z" Nov 03 20:20:19.554341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.553280 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" Nov 03 20:20:19.607500 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.607469 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85998d26-9a30-423e-91c8-0926e2f724ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85998d26-9a30-423e-91c8-0926e2f724ae-maas-api-openshift-ca\") pod \"85998d26-9a30-423e-91c8-0926e2f724ae\" (UID: \"85998d26-9a30-423e-91c8-0926e2f724ae\") " Nov 03 20:20:19.607703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.607519 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85998d26-9a30-423e-91c8-0926e2f724ae-kube-api-access-tc9sv\" (UniqueName: \"kubernetes.io/projected/85998d26-9a30-423e-91c8-0926e2f724ae-kube-api-access-tc9sv\") pod \"85998d26-9a30-423e-91c8-0926e2f724ae\" (UID: \"85998d26-9a30-423e-91c8-0926e2f724ae\") " Nov 03 20:20:19.607703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.607602 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85998d26-9a30-423e-91c8-0926e2f724ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85998d26-9a30-423e-91c8-0926e2f724ae-maas-api-xks-ca\") pod \"85998d26-9a30-423e-91c8-0926e2f724ae\" (UID: \"85998d26-9a30-423e-91c8-0926e2f724ae\") " Nov 03 20:20:19.607954 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.607926 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85998d26-9a30-423e-91c8-0926e2f724ae-maas-api-openshift-ca" pod "85998d26-9a30-423e-91c8-0926e2f724ae" (UID: "85998d26-9a30-423e-91c8-0926e2f724ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:19.608295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.608268 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85998d26-9a30-423e-91c8-0926e2f724ae-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:19.611186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.611155 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85998d26-9a30-423e-91c8-0926e2f724ae-kube-api-access-tc9sv" pod "85998d26-9a30-423e-91c8-0926e2f724ae" (UID: "85998d26-9a30-423e-91c8-0926e2f724ae"). InnerVolumeSpecName "kube-api-access-tc9sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:19.611381 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.611355 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85998d26-9a30-423e-91c8-0926e2f724ae-maas-api-xks-ca" pod "85998d26-9a30-423e-91c8-0926e2f724ae" (UID: "85998d26-9a30-423e-91c8-0926e2f724ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:19.709639 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.709605 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc9sv\" (UniqueName: \"kubernetes.io/projected/85998d26-9a30-423e-91c8-0926e2f724ae-kube-api-access-tc9sv\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:19.709639 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.709637 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85998d26-9a30-423e-91c8-0926e2f724ae-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:19.712974 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.712941 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z" Nov 03 20:20:19.834782 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:19.834679 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" podUID="6a39316d-9016-42b8-93d7-73296fd2bfde" containerName="payload-pre-processing" containerID="cri-o://9bc15d88baa04f72f3f5c7618002198a49029bc1a586788f87a96e1c7d5ed245" gracePeriod=25 Nov 03 20:20:20.125711 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.121884 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z"] Nov 03 20:20:20.156036 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:20:20.155514 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d2755a_e68d_4924_bbdd_6b0bd29d6398.slice/crio-f17140fe24bcd290b3f4d15427f33727cdbab3ffb0a588850e35529db796f54a WatchSource:0}: Error finding container f17140fe24bcd290b3f4d15427f33727cdbab3ffb0a588850e35529db796f54a: Status 404 returned error can't find the container with id f17140fe24bcd290b3f4d15427f33727cdbab3ffb0a588850e35529db796f54a Nov 03 20:20:20.290995 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.290968 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" Nov 03 20:20:20.290995 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.290989 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d" event={"ID":"85998d26-9a30-423e-91c8-0926e2f724ae","Type":"ContainerDied","Data":"4b623af387f0add41ac322d02fb4a23f0c5ca75b0618a04bb7e3a4cdd6d8df54"} Nov 03 20:20:20.291588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.291046 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b623af387f0add41ac322d02fb4a23f0c5ca75b0618a04bb7e3a4cdd6d8df54" Nov 03 20:20:20.294305 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.294277 5111 generic.go:356] "Generic (PLEG): container finished" podID="6a39316d-9016-42b8-93d7-73296fd2bfde" containerID="9bc15d88baa04f72f3f5c7618002198a49029bc1a586788f87a96e1c7d5ed245" exitCode=0 Nov 03 20:20:20.294439 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.294351 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" event={"ID":"6a39316d-9016-42b8-93d7-73296fd2bfde","Type":"ContainerDied","Data":"9bc15d88baa04f72f3f5c7618002198a49029bc1a586788f87a96e1c7d5ed245"} Nov 03 20:20:20.296594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.296569 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z" event={"ID":"30d2755a-e68d-4924-bbdd-6b0bd29d6398","Type":"ContainerStarted","Data":"f17140fe24bcd290b3f4d15427f33727cdbab3ffb0a588850e35529db796f54a"} Nov 03 20:20:20.359811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.359782 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" Nov 03 20:20:20.505927 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.505898 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a39316d-9016-42b8-93d7-73296fd2bfde-kube-api-access-7zqgz\" (UniqueName: \"kubernetes.io/projected/6a39316d-9016-42b8-93d7-73296fd2bfde-kube-api-access-7zqgz\") pod \"6a39316d-9016-42b8-93d7-73296fd2bfde\" (UID: \"6a39316d-9016-42b8-93d7-73296fd2bfde\") " Nov 03 20:20:20.506082 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.506065 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a39316d-9016-42b8-93d7-73296fd2bfde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a39316d-9016-42b8-93d7-73296fd2bfde-plugins-config-volume\") pod \"6a39316d-9016-42b8-93d7-73296fd2bfde\" (UID: \"6a39316d-9016-42b8-93d7-73296fd2bfde\") " Nov 03 20:20:20.516886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.515321 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a39316d-9016-42b8-93d7-73296fd2bfde-plugins-config-volume" pod "6a39316d-9016-42b8-93d7-73296fd2bfde" (UID: "6a39316d-9016-42b8-93d7-73296fd2bfde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:20.517157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.517127 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a39316d-9016-42b8-93d7-73296fd2bfde-kube-api-access-7zqgz" pod "6a39316d-9016-42b8-93d7-73296fd2bfde" (UID: "6a39316d-9016-42b8-93d7-73296fd2bfde"). InnerVolumeSpecName "kube-api-access-7zqgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:20.546332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.546304 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" Nov 03 20:20:20.608401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.608365 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-maas-api-openshift-ca\") pod \"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad\" (UID: \"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad\") " Nov 03 20:20:20.608634 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.608599 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-kube-api-access-49rk7\" (UniqueName: \"kubernetes.io/projected/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-kube-api-access-49rk7\") pod \"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad\" (UID: \"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad\") " Nov 03 20:20:20.608796 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.608780 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-maas-api-xks-ca\") pod \"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad\" (UID: \"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad\") " Nov 03 20:20:20.609877 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.608799 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-maas-api-openshift-ca" pod "97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad" (UID: "97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:20.609877 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.609408 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:20.609877 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.609430 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zqgz\" (UniqueName: \"kubernetes.io/projected/6a39316d-9016-42b8-93d7-73296fd2bfde-kube-api-access-7zqgz\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:20.609877 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.609448 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a39316d-9016-42b8-93d7-73296fd2bfde-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:20.611090 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.611062 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-kube-api-access-49rk7" pod "97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad" (UID: "97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad"). InnerVolumeSpecName "kube-api-access-49rk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:20.611642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.611524 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-maas-api-xks-ca" pod "97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad" (UID: "97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:20.710991 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.710918 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49rk7\" (UniqueName: \"kubernetes.io/projected/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-kube-api-access-49rk7\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:20.710991 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:20.710950 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:21.306157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:21.306132 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" Nov 03 20:20:21.306731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:21.306171 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj" event={"ID":"97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad","Type":"ContainerDied","Data":"8f8b7f3c684a306a3f5201a8b55246dd21645881b9d9bbd04fccbda59794f753"} Nov 03 20:20:21.306731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:21.306222 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f8b7f3c684a306a3f5201a8b55246dd21645881b9d9bbd04fccbda59794f753" Nov 03 20:20:21.307733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:21.307714 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" Nov 03 20:20:21.307843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:21.307731 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx" event={"ID":"6a39316d-9016-42b8-93d7-73296fd2bfde","Type":"ContainerDied","Data":"a90f81b7bbf1c018220cdb05dc5523ea1d093f829fe032e60b9ae783ec177090"} Nov 03 20:20:21.307843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:21.307785 5111 scope.go:122] "RemoveContainer" containerID="9bc15d88baa04f72f3f5c7618002198a49029bc1a586788f87a96e1c7d5ed245" Nov 03 20:20:21.309661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:21.309636 5111 generic.go:356] "Generic (PLEG): container finished" podID="30d2755a-e68d-4924-bbdd-6b0bd29d6398" containerID="600fa3efdfd9d2cd0821e42ac7eceac3e6d3f673b80ef9cf7e35e9dc1d3392a8" exitCode=0 Nov 03 20:20:21.309785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:21.309663 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z" event={"ID":"30d2755a-e68d-4924-bbdd-6b0bd29d6398","Type":"ContainerDied","Data":"600fa3efdfd9d2cd0821e42ac7eceac3e6d3f673b80ef9cf7e35e9dc1d3392a8"} Nov 03 20:20:21.377737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:21.377508 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx"] Nov 03 20:20:21.394578 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:21.394527 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-5348ca-7f46f69df9-hftbx"] Nov 03 20:20:22.215234 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:22.213820 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a39316d-9016-42b8-93d7-73296fd2bfde" path="/var/lib/kubelet/pods/6a39316d-9016-42b8-93d7-73296fd2bfde/volumes" Nov 03 20:20:22.568406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:22.568379 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z" Nov 03 20:20:22.669134 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:22.669102 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d2755a-e68d-4924-bbdd-6b0bd29d6398-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d2755a-e68d-4924-bbdd-6b0bd29d6398-maas-api-xks-ca\") pod \"30d2755a-e68d-4924-bbdd-6b0bd29d6398\" (UID: \"30d2755a-e68d-4924-bbdd-6b0bd29d6398\") " Nov 03 20:20:22.669305 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:22.669166 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d2755a-e68d-4924-bbdd-6b0bd29d6398-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d2755a-e68d-4924-bbdd-6b0bd29d6398-maas-api-openshift-ca\") pod \"30d2755a-e68d-4924-bbdd-6b0bd29d6398\" (UID: \"30d2755a-e68d-4924-bbdd-6b0bd29d6398\") " Nov 03 20:20:22.669345 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:22.669306 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d2755a-e68d-4924-bbdd-6b0bd29d6398-kube-api-access-w2x8n\" (UniqueName: \"kubernetes.io/projected/30d2755a-e68d-4924-bbdd-6b0bd29d6398-kube-api-access-w2x8n\") pod \"30d2755a-e68d-4924-bbdd-6b0bd29d6398\" (UID: \"30d2755a-e68d-4924-bbdd-6b0bd29d6398\") " Nov 03 20:20:22.669584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:22.669561 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d2755a-e68d-4924-bbdd-6b0bd29d6398-maas-api-openshift-ca" pod "30d2755a-e68d-4924-bbdd-6b0bd29d6398" (UID: "30d2755a-e68d-4924-bbdd-6b0bd29d6398"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:22.670180 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:22.670151 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d2755a-e68d-4924-bbdd-6b0bd29d6398-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:22.671440 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:22.671413 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d2755a-e68d-4924-bbdd-6b0bd29d6398-maas-api-xks-ca" pod "30d2755a-e68d-4924-bbdd-6b0bd29d6398" (UID: "30d2755a-e68d-4924-bbdd-6b0bd29d6398"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:22.671659 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:22.671639 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d2755a-e68d-4924-bbdd-6b0bd29d6398-kube-api-access-w2x8n" pod "30d2755a-e68d-4924-bbdd-6b0bd29d6398" (UID: "30d2755a-e68d-4924-bbdd-6b0bd29d6398"). InnerVolumeSpecName "kube-api-access-w2x8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:22.772214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:22.772181 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d2755a-e68d-4924-bbdd-6b0bd29d6398-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:22.772214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:22.772210 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2x8n\" (UniqueName: \"kubernetes.io/projected/30d2755a-e68d-4924-bbdd-6b0bd29d6398-kube-api-access-w2x8n\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:23.336600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:23.336558 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z" event={"ID":"30d2755a-e68d-4924-bbdd-6b0bd29d6398","Type":"ContainerDied","Data":"f17140fe24bcd290b3f4d15427f33727cdbab3ffb0a588850e35529db796f54a"} Nov 03 20:20:23.336794 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:23.336626 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f17140fe24bcd290b3f4d15427f33727cdbab3ffb0a588850e35529db796f54a" Nov 03 20:20:23.336794 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:23.336704 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z" Nov 03 20:20:23.512398 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:23.512367 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz"] Nov 03 20:20:23.516498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:23.516473 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9b1aed-79fhz"] Nov 03 20:20:24.193220 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:24.193182 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc21eef-fb5a-4543-a596-12352ea7b48c" path="/var/lib/kubelet/pods/ecc21eef-fb5a-4543-a596-12352ea7b48c/volumes" Nov 03 20:20:24.452264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:24.452154 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" containerID="cri-o://9fb2b98610f7336e8bb0364be4765e00390cda566adecfa912dd55a987669bdf" gracePeriod=45 Nov 03 20:20:24.874217 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:24.874178 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:20:25.049296 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.049135 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-dshm\") pod \"8f264669-2468-4f77-b3a4-0eea66f35352\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " Nov 03 20:20:25.049296 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.049205 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-home\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-home\") pod \"8f264669-2468-4f77-b3a4-0eea66f35352\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " Nov 03 20:20:25.056399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.049445 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f264669-2468-4f77-b3a4-0eea66f35352-kube-api-access-kkg82\" (UniqueName: \"kubernetes.io/projected/8f264669-2468-4f77-b3a4-0eea66f35352-kube-api-access-kkg82\") pod \"8f264669-2468-4f77-b3a4-0eea66f35352\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " Nov 03 20:20:25.056399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.049485 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-tmp-dir\") pod \"8f264669-2468-4f77-b3a4-0eea66f35352\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " Nov 03 20:20:25.056399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.049517 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-model-cache\") pod \"8f264669-2468-4f77-b3a4-0eea66f35352\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " Nov 03 20:20:25.056399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.051134 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-home" pod "8f264669-2468-4f77-b3a4-0eea66f35352" (UID: "8f264669-2468-4f77-b3a4-0eea66f35352"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:25.056399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.052150 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-tmp-dir" pod "8f264669-2468-4f77-b3a4-0eea66f35352" (UID: "8f264669-2468-4f77-b3a4-0eea66f35352"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:25.056399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.052414 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-model-cache" pod "8f264669-2468-4f77-b3a4-0eea66f35352" (UID: "8f264669-2468-4f77-b3a4-0eea66f35352"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:25.056399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.052656 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f264669-2468-4f77-b3a4-0eea66f35352-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f264669-2468-4f77-b3a4-0eea66f35352-tls-certs\") pod \"8f264669-2468-4f77-b3a4-0eea66f35352\" (UID: \"8f264669-2468-4f77-b3a4-0eea66f35352\") " Nov 03 20:20:25.056399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.053428 5111 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-model-cache\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:25.056399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.053448 5111 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-home\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:25.056399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.053485 5111 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-tmp-dir\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:25.067235 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.065518 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f264669-2468-4f77-b3a4-0eea66f35352-tls-certs" pod "8f264669-2468-4f77-b3a4-0eea66f35352" (UID: "8f264669-2468-4f77-b3a4-0eea66f35352"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:25.091319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.089437 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f264669-2468-4f77-b3a4-0eea66f35352-kube-api-access-kkg82" pod "8f264669-2468-4f77-b3a4-0eea66f35352" (UID: "8f264669-2468-4f77-b3a4-0eea66f35352"). InnerVolumeSpecName "kube-api-access-kkg82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:25.091319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.090110 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-dshm" pod "8f264669-2468-4f77-b3a4-0eea66f35352" (UID: "8f264669-2468-4f77-b3a4-0eea66f35352"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:25.155547 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.155495 5111 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f264669-2468-4f77-b3a4-0eea66f35352-tls-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:25.155547 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.155542 5111 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f264669-2468-4f77-b3a4-0eea66f35352-dshm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:25.155744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.155559 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkg82\" (UniqueName: \"kubernetes.io/projected/8f264669-2468-4f77-b3a4-0eea66f35352-kube-api-access-kkg82\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:25.362630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.359043 5111 generic.go:356] "Generic (PLEG): container finished" podID="8f264669-2468-4f77-b3a4-0eea66f35352" containerID="9fb2b98610f7336e8bb0364be4765e00390cda566adecfa912dd55a987669bdf" exitCode=0 Nov 03 20:20:25.362630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.359100 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" Nov 03 20:20:25.362630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.359134 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" event={"ID":"8f264669-2468-4f77-b3a4-0eea66f35352","Type":"ContainerDied","Data":"9fb2b98610f7336e8bb0364be4765e00390cda566adecfa912dd55a987669bdf"} Nov 03 20:20:25.362630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.359192 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd" event={"ID":"8f264669-2468-4f77-b3a4-0eea66f35352","Type":"ContainerDied","Data":"21cb62ac8284af04cb9175256dd13fd95a9605e5ec29f53e1b3928af9ed5adf6"} Nov 03 20:20:25.362630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.359214 5111 scope.go:122] "RemoveContainer" containerID="9fb2b98610f7336e8bb0364be4765e00390cda566adecfa912dd55a987669bdf" Nov 03 20:20:25.376996 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.376488 5111 scope.go:122] "RemoveContainer" containerID="9fb2b98610f7336e8bb0364be4765e00390cda566adecfa912dd55a987669bdf" Nov 03 20:20:25.377569 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:20:25.377487 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb2b98610f7336e8bb0364be4765e00390cda566adecfa912dd55a987669bdf\": container with ID starting with 9fb2b98610f7336e8bb0364be4765e00390cda566adecfa912dd55a987669bdf not found: ID does not exist" containerID="9fb2b98610f7336e8bb0364be4765e00390cda566adecfa912dd55a987669bdf" Nov 03 20:20:25.382001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.377703 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb2b98610f7336e8bb0364be4765e00390cda566adecfa912dd55a987669bdf"} err="failed to get container status \"9fb2b98610f7336e8bb0364be4765e00390cda566adecfa912dd55a987669bdf\": rpc error: code = NotFound desc = could not find container \"9fb2b98610f7336e8bb0364be4765e00390cda566adecfa912dd55a987669bdf\": container with ID starting with 9fb2b98610f7336e8bb0364be4765e00390cda566adecfa912dd55a987669bdf not found: ID does not exist" Nov 03 20:20:25.411196 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.411158 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd"] Nov 03 20:20:25.411371 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:25.411219 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/distinct-w4-5348ca-kserve-7d9dfdc869-lxfpd"] Nov 03 20:20:26.221569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:26.219974 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" path="/var/lib/kubelet/pods/8f264669-2468-4f77-b3a4-0eea66f35352/volumes" Nov 03 20:20:29.483937 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:29.483899 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq"] Nov 03 20:20:29.501964 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:29.501928 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz"] Nov 03 20:20:30.511425 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:30.511328 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw"] Nov 03 20:20:30.511847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:30.511743 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" podUID="e140c17f-f261-4a57-938d-7199ddcce0a3" containerName="istio-proxy" containerID="cri-o://0edd1061ffc6c63e0cb8ed0b28562f31af788220f8e2505de1f96d5e2e1a7ed8" gracePeriod=30 Nov 03 20:20:32.417605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:32.417571 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js"] Nov 03 20:20:32.423598 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:32.423416 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5348ca-mz2js"] Nov 03 20:20:34.187764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:34.187726 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ef63c4d-f39d-4623-869c-26983ba16e10" path="/var/lib/kubelet/pods/7ef63c4d-f39d-4623-869c-26983ba16e10/volumes" Nov 03 20:20:34.485110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:34.485027 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" podUID="f6017f42-4ca1-4feb-a33a-c2eaf1585d8d" containerName="payload-processing" containerID="cri-o://a2c8a83ad7e0d0f5ae42c85e4b22c62f9424cb7b779f58bccbbc4335aa0793fc" gracePeriod=25 Nov 03 20:20:34.503148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:34.503105 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" podUID="eff84f5c-617a-4dcf-8bda-ba203f11f648" containerName="payload-pre-processing" containerID="cri-o://7186bc456140056b5b92e4e24c67de4757116df8cebb98d04151e646ecfe08b5" gracePeriod=25 Nov 03 20:20:35.074282 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:20:35.074234 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:20:35.998402 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:35.998353 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z"] Nov 03 20:20:36.010209 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.010174 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-babea4-9jm7z"] Nov 03 20:20:36.309235 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.309200 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d2755a-e68d-4924-bbdd-6b0bd29d6398" path="/var/lib/kubelet/pods/30d2755a-e68d-4924-bbdd-6b0bd29d6398/volumes" Nov 03 20:20:36.392632 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.389737 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:20:36.429032 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.428997 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-envoy\") pod \"e140c17f-f261-4a57-938d-7199ddcce0a3\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " Nov 03 20:20:36.429769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.429745 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e140c17f-f261-4a57-938d-7199ddcce0a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e140c17f-f261-4a57-938d-7199ddcce0a3-istiod-ca-cert\") pod \"e140c17f-f261-4a57-938d-7199ddcce0a3\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " Nov 03 20:20:36.430398 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.430288 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e140c17f-f261-4a57-938d-7199ddcce0a3-istiod-ca-cert" pod "e140c17f-f261-4a57-938d-7199ddcce0a3" (UID: "e140c17f-f261-4a57-938d-7199ddcce0a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:36.430579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.430562 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-workload-certs\") pod \"e140c17f-f261-4a57-938d-7199ddcce0a3\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " Nov 03 20:20:36.431067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.431047 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e140c17f-f261-4a57-938d-7199ddcce0a3-kube-api-access-djjvb\" (UniqueName: \"kubernetes.io/projected/e140c17f-f261-4a57-938d-7199ddcce0a3-kube-api-access-djjvb\") pod \"e140c17f-f261-4a57-938d-7199ddcce0a3\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " Nov 03 20:20:36.431331 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.431313 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-token\" (UniqueName: \"kubernetes.io/projected/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-token\") pod \"e140c17f-f261-4a57-938d-7199ddcce0a3\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " Nov 03 20:20:36.432030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.432013 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-workload-socket\") pod \"e140c17f-f261-4a57-938d-7199ddcce0a3\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " Nov 03 20:20:36.432475 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.432459 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-credential-socket\") pod \"e140c17f-f261-4a57-938d-7199ddcce0a3\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " Nov 03 20:20:36.432942 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.432924 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-podinfo\") pod \"e140c17f-f261-4a57-938d-7199ddcce0a3\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " Nov 03 20:20:36.433624 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.433603 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-data\") pod \"e140c17f-f261-4a57-938d-7199ddcce0a3\" (UID: \"e140c17f-f261-4a57-938d-7199ddcce0a3\") " Nov 03 20:20:36.434915 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.434892 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e140c17f-f261-4a57-938d-7199ddcce0a3-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:36.435491 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.431954 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-workload-certs" pod "e140c17f-f261-4a57-938d-7199ddcce0a3" (UID: "e140c17f-f261-4a57-938d-7199ddcce0a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:36.435668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.432389 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-workload-socket" pod "e140c17f-f261-4a57-938d-7199ddcce0a3" (UID: "e140c17f-f261-4a57-938d-7199ddcce0a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:36.435771 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.432850 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-credential-socket" pod "e140c17f-f261-4a57-938d-7199ddcce0a3" (UID: "e140c17f-f261-4a57-938d-7199ddcce0a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:36.435868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.433983 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-data" pod "e140c17f-f261-4a57-938d-7199ddcce0a3" (UID: "e140c17f-f261-4a57-938d-7199ddcce0a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:36.463030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.462973 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e140c17f-f261-4a57-938d-7199ddcce0a3-kube-api-access-djjvb" pod "e140c17f-f261-4a57-938d-7199ddcce0a3" (UID: "e140c17f-f261-4a57-938d-7199ddcce0a3"). InnerVolumeSpecName "kube-api-access-djjvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:36.463963 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.463902 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-podinfo" pod "e140c17f-f261-4a57-938d-7199ddcce0a3" (UID: "e140c17f-f261-4a57-938d-7199ddcce0a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:20:36.464232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.464202 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-token" pod "e140c17f-f261-4a57-938d-7199ddcce0a3" (UID: "e140c17f-f261-4a57-938d-7199ddcce0a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:36.470997 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.470957 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-envoy" pod "e140c17f-f261-4a57-938d-7199ddcce0a3" (UID: "e140c17f-f261-4a57-938d-7199ddcce0a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:36.543884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.543554 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:36.543884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.543589 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:36.543884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.543603 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:36.543884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.543704 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:36.543884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.543718 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:36.543884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.543730 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:36.543884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.543745 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e140c17f-f261-4a57-938d-7199ddcce0a3-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:36.543884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.543758 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djjvb\" (UniqueName: \"kubernetes.io/projected/e140c17f-f261-4a57-938d-7199ddcce0a3-kube-api-access-djjvb\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:36.559439 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.559394 5111 generic.go:356] "Generic (PLEG): container finished" podID="e140c17f-f261-4a57-938d-7199ddcce0a3" containerID="0edd1061ffc6c63e0cb8ed0b28562f31af788220f8e2505de1f96d5e2e1a7ed8" exitCode=0 Nov 03 20:20:36.559637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.559549 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" event={"ID":"e140c17f-f261-4a57-938d-7199ddcce0a3","Type":"ContainerDied","Data":"0edd1061ffc6c63e0cb8ed0b28562f31af788220f8e2505de1f96d5e2e1a7ed8"} Nov 03 20:20:36.559637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.559603 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" event={"ID":"e140c17f-f261-4a57-938d-7199ddcce0a3","Type":"ContainerDied","Data":"1f6e008791d37027a97a71d1ad58ff42448601c048470953cb5093eae15d04b3"} Nov 03 20:20:36.559637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.559632 5111 scope.go:122] "RemoveContainer" containerID="0edd1061ffc6c63e0cb8ed0b28562f31af788220f8e2505de1f96d5e2e1a7ed8" Nov 03 20:20:36.559822 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.559809 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw" Nov 03 20:20:36.581708 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.581676 5111 scope.go:122] "RemoveContainer" containerID="437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40" Nov 03 20:20:36.695150 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.695039 5111 scope.go:122] "RemoveContainer" containerID="437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40" Nov 03 20:20:36.697561 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:20:36.695650 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40\": container with ID starting with 437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40 not found: ID does not exist" containerID="437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40" Nov 03 20:20:36.697561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.695689 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40"} err="failed to get container status \"437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40\": rpc error: code = NotFound desc = could not find container \"437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40\": container with ID starting with 437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40 not found: ID does not exist" Nov 03 20:20:36.697561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.695715 5111 scope.go:122] "RemoveContainer" containerID="0edd1061ffc6c63e0cb8ed0b28562f31af788220f8e2505de1f96d5e2e1a7ed8" Nov 03 20:20:36.697561 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:20:36.696037 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0edd1061ffc6c63e0cb8ed0b28562f31af788220f8e2505de1f96d5e2e1a7ed8\": container with ID starting with 0edd1061ffc6c63e0cb8ed0b28562f31af788220f8e2505de1f96d5e2e1a7ed8 not found: ID does not exist" containerID="0edd1061ffc6c63e0cb8ed0b28562f31af788220f8e2505de1f96d5e2e1a7ed8" Nov 03 20:20:36.697561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.696066 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0edd1061ffc6c63e0cb8ed0b28562f31af788220f8e2505de1f96d5e2e1a7ed8"} err="failed to get container status \"0edd1061ffc6c63e0cb8ed0b28562f31af788220f8e2505de1f96d5e2e1a7ed8\": rpc error: code = NotFound desc = could not find container \"0edd1061ffc6c63e0cb8ed0b28562f31af788220f8e2505de1f96d5e2e1a7ed8\": container with ID starting with 0edd1061ffc6c63e0cb8ed0b28562f31af788220f8e2505de1f96d5e2e1a7ed8 not found: ID does not exist" Nov 03 20:20:36.697561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.696087 5111 scope.go:122] "RemoveContainer" containerID="437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40" Nov 03 20:20:36.697561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.696345 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40"} err="failed to get container status \"437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40\": rpc error: code = NotFound desc = could not find container \"437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40\": container with ID starting with 437695ae76aea892b7cd0fb2cddd1aca8ba75b787c576fd625c04fdeb6658b40 not found: ID does not exist" Nov 03 20:20:36.702552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.702126 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw"] Nov 03 20:20:36.713421 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:36.713374 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9b1aed-openshift-default-69b65857d9-q4pvw"] Nov 03 20:20:37.186591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:37.184175 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" podUID="f6017f42-4ca1-4feb-a33a-c2eaf1585d8d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:20:37.233564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:37.232946 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" podUID="eff84f5c-617a-4dcf-8bda-ba203f11f648" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:20:38.196889 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:38.196858 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e140c17f-f261-4a57-938d-7199ddcce0a3" path="/var/lib/kubelet/pods/e140c17f-f261-4a57-938d-7199ddcce0a3/volumes" Nov 03 20:20:40.428284 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:40.428243 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch"] Nov 03 20:20:40.428975 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:40.428503 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" podUID="330ae291-dce5-435f-aac6-5cbc4214c273" containerName="istio-proxy" containerID="cri-o://d9aa98982d2c7b6b3083db68240b1f078c8e6492075c0648a8fb82236fbf31db" gracePeriod=30 Nov 03 20:20:40.440346 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:40.440306 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q"] Nov 03 20:20:40.440597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:40.440568 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" podUID="653d951c-536d-405b-bc11-c9985924b511" containerName="istio-proxy" containerID="cri-o://339047ed53a74a05fb461cae65cccaa9ad55246a4199addf4b697d1e56fda5c8" gracePeriod=30 Nov 03 20:20:43.902971 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.902872 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f"] Nov 03 20:20:43.904120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904082 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6a39316d-9016-42b8-93d7-73296fd2bfde" containerName="payload-pre-processing" Nov 03 20:20:43.904276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904154 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a39316d-9016-42b8-93d7-73296fd2bfde" containerName="payload-pre-processing" Nov 03 20:20:43.904276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904183 5111 cpu_manager.go:414] "containerMap: removing container" podUID="85998d26-9a30-423e-91c8-0926e2f724ae" containerName="revoke-keys" Nov 03 20:20:43.904276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904193 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85998d26-9a30-423e-91c8-0926e2f724ae" containerName="revoke-keys" Nov 03 20:20:43.904276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904225 5111 cpu_manager.go:414] "containerMap: removing container" podUID="30d2755a-e68d-4924-bbdd-6b0bd29d6398" containerName="revoke-keys" Nov 03 20:20:43.904276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904232 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d2755a-e68d-4924-bbdd-6b0bd29d6398" containerName="revoke-keys" Nov 03 20:20:43.904276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904248 5111 cpu_manager.go:414] "containerMap: removing container" podUID="97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad" containerName="revoke-keys" Nov 03 20:20:43.904276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904255 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad" containerName="revoke-keys" Nov 03 20:20:43.904713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904288 5111 cpu_manager.go:414] "containerMap: removing container" podUID="e140c17f-f261-4a57-938d-7199ddcce0a3" containerName="istio-proxy" Nov 03 20:20:43.904713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904298 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e140c17f-f261-4a57-938d-7199ddcce0a3" containerName="istio-proxy" Nov 03 20:20:43.904713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904313 5111 cpu_manager.go:414] "containerMap: removing container" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" Nov 03 20:20:43.904713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904320 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" Nov 03 20:20:43.904713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904358 5111 cpu_manager.go:414] "containerMap: removing container" podUID="e140c17f-f261-4a57-938d-7199ddcce0a3" containerName="istio-proxy" Nov 03 20:20:43.904713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904367 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e140c17f-f261-4a57-938d-7199ddcce0a3" containerName="istio-proxy" Nov 03 20:20:43.904713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904585 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d2755a-e68d-4924-bbdd-6b0bd29d6398" containerName="revoke-keys" Nov 03 20:20:43.904713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904606 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad" containerName="revoke-keys" Nov 03 20:20:43.904713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904620 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="e140c17f-f261-4a57-938d-7199ddcce0a3" containerName="istio-proxy" Nov 03 20:20:43.904713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904629 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f264669-2468-4f77-b3a4-0eea66f35352" containerName="main" Nov 03 20:20:43.904713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904648 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a39316d-9016-42b8-93d7-73296fd2bfde" containerName="payload-pre-processing" Nov 03 20:20:43.904713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.904660 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="85998d26-9a30-423e-91c8-0926e2f724ae" containerName="revoke-keys" Nov 03 20:20:43.912031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.912005 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:43.921675 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.921199 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-456f55-openshift-default-dockercfg-8jtw5\"" Nov 03 20:20:43.942969 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:43.942936 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f"] Nov 03 20:20:44.020556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.016425 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-workload-socket\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.020556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.016508 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-envoy\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.052778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.021026 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-podinfo\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.052778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.021122 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5478802b-2542-47ee-919d-5a8ad79a5e6a-istiod-ca-cert\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.052778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.021221 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-credential-socket\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.052778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.021327 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-workload-certs\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.052778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.021400 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmsqf\" (UniqueName: \"kubernetes.io/projected/5478802b-2542-47ee-919d-5a8ad79a5e6a-kube-api-access-mmsqf\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.052778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.021549 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-data\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.052778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.021640 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-token\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.155607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.124773 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-envoy\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.155607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.124889 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-podinfo\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.155607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.124942 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5478802b-2542-47ee-919d-5a8ad79a5e6a-istiod-ca-cert\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.155607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.125032 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-credential-socket\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.155607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.125126 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-workload-certs\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.155607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.125186 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmsqf\" (UniqueName: \"kubernetes.io/projected/5478802b-2542-47ee-919d-5a8ad79a5e6a-kube-api-access-mmsqf\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.155607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.125313 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-data\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.155607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.125438 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-token\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.155607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.125578 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-workload-socket\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.155607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.126207 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-workload-socket\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.166999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.158961 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-workload-certs\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.166999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.159975 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-data\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.166999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.161204 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5478802b-2542-47ee-919d-5a8ad79a5e6a-istiod-ca-cert\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.166999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.161796 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-credential-socket\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.268478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.268379 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-podinfo\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.275608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.275207 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-token\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.278496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.276884 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-envoy\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.286991 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.286904 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmsqf\" (UniqueName: \"kubernetes.io/projected/5478802b-2542-47ee-919d-5a8ad79a5e6a-kube-api-access-mmsqf\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.528760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.528676 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:44.827114 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.827080 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f"] Nov 03 20:20:44.827365 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:20:44.827339 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5478802b_2542_47ee_919d_5a8ad79a5e6a.slice/crio-8db3373dd0b90cf3383d55f563683c8c450d0220d9cc3131c93f141e4d4f554e WatchSource:0}: Error finding container 8db3373dd0b90cf3383d55f563683c8c450d0220d9cc3131c93f141e4d4f554e: Status 404 returned error can't find the container with id 8db3373dd0b90cf3383d55f563683c8c450d0220d9cc3131c93f141e4d4f554e Nov 03 20:20:44.830181 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.830146 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:20:44.830291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.830226 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:20:44.830291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.830274 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:20:44.954480 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:44.953684 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" event={"ID":"5478802b-2542-47ee-919d-5a8ad79a5e6a","Type":"ContainerStarted","Data":"8db3373dd0b90cf3383d55f563683c8c450d0220d9cc3131c93f141e4d4f554e"} Nov 03 20:20:45.572609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.572567 5111 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 20:20:45.572803 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.572641 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" podUID="653d951c-536d-405b-bc11-c9985924b511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 20:20:45.751792 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.751763 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:20:45.782431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.782397 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330ae291-dce5-435f-aac6-5cbc4214c273-istio-token\" (UniqueName: \"kubernetes.io/projected/330ae291-dce5-435f-aac6-5cbc4214c273-istio-token\") pod \"330ae291-dce5-435f-aac6-5cbc4214c273\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " Nov 03 20:20:45.782645 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.782525 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-workload-certs\") pod \"330ae291-dce5-435f-aac6-5cbc4214c273\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " Nov 03 20:20:45.782645 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.782607 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/330ae291-dce5-435f-aac6-5cbc4214c273-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330ae291-dce5-435f-aac6-5cbc4214c273-istiod-ca-cert\") pod \"330ae291-dce5-435f-aac6-5cbc4214c273\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " Nov 03 20:20:45.782781 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.782668 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330ae291-dce5-435f-aac6-5cbc4214c273-kube-api-access-2zdfl\" (UniqueName: \"kubernetes.io/projected/330ae291-dce5-435f-aac6-5cbc4214c273-kube-api-access-2zdfl\") pod \"330ae291-dce5-435f-aac6-5cbc4214c273\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " Nov 03 20:20:45.782837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.782782 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-istio-envoy\") pod \"330ae291-dce5-435f-aac6-5cbc4214c273\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " Nov 03 20:20:45.784008 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.782941 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-istio-data\") pod \"330ae291-dce5-435f-aac6-5cbc4214c273\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " Nov 03 20:20:45.784008 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.782976 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-workload-socket\") pod \"330ae291-dce5-435f-aac6-5cbc4214c273\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " Nov 03 20:20:45.784008 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.783018 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-credential-socket\") pod \"330ae291-dce5-435f-aac6-5cbc4214c273\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " Nov 03 20:20:45.784008 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.783091 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/330ae291-dce5-435f-aac6-5cbc4214c273-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330ae291-dce5-435f-aac6-5cbc4214c273-istio-podinfo\") pod \"330ae291-dce5-435f-aac6-5cbc4214c273\" (UID: \"330ae291-dce5-435f-aac6-5cbc4214c273\") " Nov 03 20:20:45.784008 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.783306 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330ae291-dce5-435f-aac6-5cbc4214c273-istiod-ca-cert" pod "330ae291-dce5-435f-aac6-5cbc4214c273" (UID: "330ae291-dce5-435f-aac6-5cbc4214c273"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:45.784008 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.783849 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330ae291-dce5-435f-aac6-5cbc4214c273-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.784008 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.783878 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-workload-certs" pod "330ae291-dce5-435f-aac6-5cbc4214c273" (UID: "330ae291-dce5-435f-aac6-5cbc4214c273"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:45.784008 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.783917 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-istio-data" pod "330ae291-dce5-435f-aac6-5cbc4214c273" (UID: "330ae291-dce5-435f-aac6-5cbc4214c273"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:45.784527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.784372 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-credential-socket" pod "330ae291-dce5-435f-aac6-5cbc4214c273" (UID: "330ae291-dce5-435f-aac6-5cbc4214c273"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:45.784527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.784373 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-workload-socket" pod "330ae291-dce5-435f-aac6-5cbc4214c273" (UID: "330ae291-dce5-435f-aac6-5cbc4214c273"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:45.785294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.785267 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330ae291-dce5-435f-aac6-5cbc4214c273-istio-token" pod "330ae291-dce5-435f-aac6-5cbc4214c273" (UID: "330ae291-dce5-435f-aac6-5cbc4214c273"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:45.785910 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.785885 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330ae291-dce5-435f-aac6-5cbc4214c273-kube-api-access-2zdfl" pod "330ae291-dce5-435f-aac6-5cbc4214c273" (UID: "330ae291-dce5-435f-aac6-5cbc4214c273"). InnerVolumeSpecName "kube-api-access-2zdfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:45.786753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.786733 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/330ae291-dce5-435f-aac6-5cbc4214c273-istio-podinfo" pod "330ae291-dce5-435f-aac6-5cbc4214c273" (UID: "330ae291-dce5-435f-aac6-5cbc4214c273"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:20:45.787039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.787021 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-istio-envoy" pod "330ae291-dce5-435f-aac6-5cbc4214c273" (UID: "330ae291-dce5-435f-aac6-5cbc4214c273"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:45.788190 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.788168 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:20:45.884782 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.884737 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-istio-data\") pod \"653d951c-536d-405b-bc11-c9985924b511\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " Nov 03 20:20:45.884992 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.884846 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653d951c-536d-405b-bc11-c9985924b511-istio-token\" (UniqueName: \"kubernetes.io/projected/653d951c-536d-405b-bc11-c9985924b511-istio-token\") pod \"653d951c-536d-405b-bc11-c9985924b511\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " Nov 03 20:20:45.884992 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.884963 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-istio-envoy\") pod \"653d951c-536d-405b-bc11-c9985924b511\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " Nov 03 20:20:45.885124 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.885053 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-workload-socket\") pod \"653d951c-536d-405b-bc11-c9985924b511\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " Nov 03 20:20:45.885181 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.885117 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/653d951c-536d-405b-bc11-c9985924b511-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653d951c-536d-405b-bc11-c9985924b511-istio-podinfo\") pod \"653d951c-536d-405b-bc11-c9985924b511\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " Nov 03 20:20:45.885181 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.885160 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653d951c-536d-405b-bc11-c9985924b511-kube-api-access-9w7zp\" (UniqueName: \"kubernetes.io/projected/653d951c-536d-405b-bc11-c9985924b511-kube-api-access-9w7zp\") pod \"653d951c-536d-405b-bc11-c9985924b511\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " Nov 03 20:20:45.885281 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.885196 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653d951c-536d-405b-bc11-c9985924b511-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653d951c-536d-405b-bc11-c9985924b511-istiod-ca-cert\") pod \"653d951c-536d-405b-bc11-c9985924b511\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " Nov 03 20:20:45.885281 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.885256 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-credential-socket\") pod \"653d951c-536d-405b-bc11-c9985924b511\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " Nov 03 20:20:45.885393 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.885382 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-workload-certs\") pod \"653d951c-536d-405b-bc11-c9985924b511\" (UID: \"653d951c-536d-405b-bc11-c9985924b511\") " Nov 03 20:20:45.886622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.885526 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-istio-data" pod "653d951c-536d-405b-bc11-c9985924b511" (UID: "653d951c-536d-405b-bc11-c9985924b511"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:45.886622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.886159 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330ae291-dce5-435f-aac6-5cbc4214c273-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.886622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.886183 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.886622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.886182 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653d951c-536d-405b-bc11-c9985924b511-istiod-ca-cert" pod "653d951c-536d-405b-bc11-c9985924b511" (UID: "653d951c-536d-405b-bc11-c9985924b511"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:45.886622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.886200 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zdfl\" (UniqueName: \"kubernetes.io/projected/330ae291-dce5-435f-aac6-5cbc4214c273-kube-api-access-2zdfl\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.886622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.886215 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.886622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.886230 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.886622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.886245 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.886622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.886259 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ae291-dce5-435f-aac6-5cbc4214c273-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.886622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.886272 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.886622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.886286 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330ae291-dce5-435f-aac6-5cbc4214c273-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.886622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.886554 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-workload-certs" pod "653d951c-536d-405b-bc11-c9985924b511" (UID: "653d951c-536d-405b-bc11-c9985924b511"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:45.887224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.886803 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-credential-socket" pod "653d951c-536d-405b-bc11-c9985924b511" (UID: "653d951c-536d-405b-bc11-c9985924b511"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:45.887718 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.887687 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-workload-socket" pod "653d951c-536d-405b-bc11-c9985924b511" (UID: "653d951c-536d-405b-bc11-c9985924b511"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:45.888092 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.888066 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/653d951c-536d-405b-bc11-c9985924b511-istio-podinfo" pod "653d951c-536d-405b-bc11-c9985924b511" (UID: "653d951c-536d-405b-bc11-c9985924b511"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:20:45.889092 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.889071 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-istio-envoy" pod "653d951c-536d-405b-bc11-c9985924b511" (UID: "653d951c-536d-405b-bc11-c9985924b511"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:45.889279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.889251 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653d951c-536d-405b-bc11-c9985924b511-kube-api-access-9w7zp" pod "653d951c-536d-405b-bc11-c9985924b511" (UID: "653d951c-536d-405b-bc11-c9985924b511"). InnerVolumeSpecName "kube-api-access-9w7zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:45.889331 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.889291 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653d951c-536d-405b-bc11-c9985924b511-istio-token" pod "653d951c-536d-405b-bc11-c9985924b511" (UID: "653d951c-536d-405b-bc11-c9985924b511"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:45.963962 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.963855 5111 generic.go:356] "Generic (PLEG): container finished" podID="653d951c-536d-405b-bc11-c9985924b511" containerID="339047ed53a74a05fb461cae65cccaa9ad55246a4199addf4b697d1e56fda5c8" exitCode=0 Nov 03 20:20:45.963962 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.963904 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" Nov 03 20:20:45.963962 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.963948 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" event={"ID":"653d951c-536d-405b-bc11-c9985924b511","Type":"ContainerDied","Data":"339047ed53a74a05fb461cae65cccaa9ad55246a4199addf4b697d1e56fda5c8"} Nov 03 20:20:45.964670 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.964005 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q" event={"ID":"653d951c-536d-405b-bc11-c9985924b511","Type":"ContainerDied","Data":"34de2bcea34147d5bff36f28aa9e9ef23e5132aa99e88fa95c1406d6ae76198a"} Nov 03 20:20:45.964670 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.964074 5111 scope.go:122] "RemoveContainer" containerID="339047ed53a74a05fb461cae65cccaa9ad55246a4199addf4b697d1e56fda5c8" Nov 03 20:20:45.966205 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.966177 5111 generic.go:356] "Generic (PLEG): container finished" podID="330ae291-dce5-435f-aac6-5cbc4214c273" containerID="d9aa98982d2c7b6b3083db68240b1f078c8e6492075c0648a8fb82236fbf31db" exitCode=0 Nov 03 20:20:45.966426 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.966404 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" event={"ID":"330ae291-dce5-435f-aac6-5cbc4214c273","Type":"ContainerDied","Data":"d9aa98982d2c7b6b3083db68240b1f078c8e6492075c0648a8fb82236fbf31db"} Nov 03 20:20:45.966625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.966456 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" event={"ID":"330ae291-dce5-435f-aac6-5cbc4214c273","Type":"ContainerDied","Data":"a3c54643ca7fb49071443e186458da38fd4d77253fccd1de4f56a5ef5f816d1d"} Nov 03 20:20:45.966625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.966528 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch" Nov 03 20:20:45.969778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.969754 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" event={"ID":"5478802b-2542-47ee-919d-5a8ad79a5e6a","Type":"ContainerStarted","Data":"041b235c029964f198d7c3e2af1e79a8f57e3c3b45d4528370b72a2cac25709b"} Nov 03 20:20:45.979857 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.979831 5111 scope.go:122] "RemoveContainer" containerID="339047ed53a74a05fb461cae65cccaa9ad55246a4199addf4b697d1e56fda5c8" Nov 03 20:20:45.980238 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:20:45.980215 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339047ed53a74a05fb461cae65cccaa9ad55246a4199addf4b697d1e56fda5c8\": container with ID starting with 339047ed53a74a05fb461cae65cccaa9ad55246a4199addf4b697d1e56fda5c8 not found: ID does not exist" containerID="339047ed53a74a05fb461cae65cccaa9ad55246a4199addf4b697d1e56fda5c8" Nov 03 20:20:45.980337 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.980248 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339047ed53a74a05fb461cae65cccaa9ad55246a4199addf4b697d1e56fda5c8"} err="failed to get container status \"339047ed53a74a05fb461cae65cccaa9ad55246a4199addf4b697d1e56fda5c8\": rpc error: code = NotFound desc = could not find container \"339047ed53a74a05fb461cae65cccaa9ad55246a4199addf4b697d1e56fda5c8\": container with ID starting with 339047ed53a74a05fb461cae65cccaa9ad55246a4199addf4b697d1e56fda5c8 not found: ID does not exist" Nov 03 20:20:45.980337 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.980271 5111 scope.go:122] "RemoveContainer" containerID="d9aa98982d2c7b6b3083db68240b1f078c8e6492075c0648a8fb82236fbf31db" Nov 03 20:20:45.987625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.987598 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.987737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.987627 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.987737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.987641 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653d951c-536d-405b-bc11-c9985924b511-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.987737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.987654 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w7zp\" (UniqueName: \"kubernetes.io/projected/653d951c-536d-405b-bc11-c9985924b511-kube-api-access-9w7zp\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.987737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.987667 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653d951c-536d-405b-bc11-c9985924b511-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.987737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.987679 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.987737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.987691 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653d951c-536d-405b-bc11-c9985924b511-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.987737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.987702 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653d951c-536d-405b-bc11-c9985924b511-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:45.997351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.997327 5111 scope.go:122] "RemoveContainer" containerID="d9aa98982d2c7b6b3083db68240b1f078c8e6492075c0648a8fb82236fbf31db" Nov 03 20:20:45.997737 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:20:45.997715 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9aa98982d2c7b6b3083db68240b1f078c8e6492075c0648a8fb82236fbf31db\": container with ID starting with d9aa98982d2c7b6b3083db68240b1f078c8e6492075c0648a8fb82236fbf31db not found: ID does not exist" containerID="d9aa98982d2c7b6b3083db68240b1f078c8e6492075c0648a8fb82236fbf31db" Nov 03 20:20:45.997819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:45.997751 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9aa98982d2c7b6b3083db68240b1f078c8e6492075c0648a8fb82236fbf31db"} err="failed to get container status \"d9aa98982d2c7b6b3083db68240b1f078c8e6492075c0648a8fb82236fbf31db\": rpc error: code = NotFound desc = could not find container \"d9aa98982d2c7b6b3083db68240b1f078c8e6492075c0648a8fb82236fbf31db\": container with ID starting with d9aa98982d2c7b6b3083db68240b1f078c8e6492075c0648a8fb82236fbf31db not found: ID does not exist" Nov 03 20:20:46.007754 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:46.007699 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" podStartSLOduration=3.00768233 podStartE2EDuration="3.00768233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:46.002379205 +0000 UTC m=+2475.964757645" watchObservedRunningTime="2026-11-03 20:20:46.00768233 +0000 UTC m=+2475.970060769" Nov 03 20:20:46.020916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:46.020878 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch"] Nov 03 20:20:46.027885 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:46.027844 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-vnfch"] Nov 03 20:20:46.041640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:46.041611 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q"] Nov 03 20:20:46.051571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:46.051521 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-w2x8q"] Nov 03 20:20:46.187253 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:46.187217 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330ae291-dce5-435f-aac6-5cbc4214c273" path="/var/lib/kubelet/pods/330ae291-dce5-435f-aac6-5cbc4214c273/volumes" Nov 03 20:20:46.187882 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:46.187852 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653d951c-536d-405b-bc11-c9985924b511" path="/var/lib/kubelet/pods/653d951c-536d-405b-bc11-c9985924b511/volumes" Nov 03 20:20:46.529051 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:46.529017 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:46.537355 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:46.537328 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:46.983088 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:46.983056 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:46.985685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:46.984091 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:20:47.092630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:47.092585 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" podUID="eff84f5c-617a-4dcf-8bda-ba203f11f648" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:20:47.093068 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:47.093033 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" podUID="f6017f42-4ca1-4feb-a33a-c2eaf1585d8d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:20:54.857507 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:54.857466 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" Nov 03 20:20:54.893086 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:54.893052 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" Nov 03 20:20:54.937382 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:54.937342 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff84f5c-617a-4dcf-8bda-ba203f11f648-kube-api-access-d4hmj\" (UniqueName: \"kubernetes.io/projected/eff84f5c-617a-4dcf-8bda-ba203f11f648-kube-api-access-d4hmj\") pod \"eff84f5c-617a-4dcf-8bda-ba203f11f648\" (UID: \"eff84f5c-617a-4dcf-8bda-ba203f11f648\") " Nov 03 20:20:54.937595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:54.937412 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6017f42-4ca1-4feb-a33a-c2eaf1585d8d-kube-api-access-h7xf8\" (UniqueName: \"kubernetes.io/projected/f6017f42-4ca1-4feb-a33a-c2eaf1585d8d-kube-api-access-h7xf8\") pod \"f6017f42-4ca1-4feb-a33a-c2eaf1585d8d\" (UID: \"f6017f42-4ca1-4feb-a33a-c2eaf1585d8d\") " Nov 03 20:20:54.937595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:54.937585 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6017f42-4ca1-4feb-a33a-c2eaf1585d8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6017f42-4ca1-4feb-a33a-c2eaf1585d8d-plugins-config-volume\") pod \"f6017f42-4ca1-4feb-a33a-c2eaf1585d8d\" (UID: \"f6017f42-4ca1-4feb-a33a-c2eaf1585d8d\") " Nov 03 20:20:54.937771 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:54.937745 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eff84f5c-617a-4dcf-8bda-ba203f11f648-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff84f5c-617a-4dcf-8bda-ba203f11f648-plugins-config-volume\") pod \"eff84f5c-617a-4dcf-8bda-ba203f11f648\" (UID: \"eff84f5c-617a-4dcf-8bda-ba203f11f648\") " Nov 03 20:20:54.938472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:54.938186 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6017f42-4ca1-4feb-a33a-c2eaf1585d8d-plugins-config-volume" pod "f6017f42-4ca1-4feb-a33a-c2eaf1585d8d" (UID: "f6017f42-4ca1-4feb-a33a-c2eaf1585d8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:54.938472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:54.938271 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff84f5c-617a-4dcf-8bda-ba203f11f648-plugins-config-volume" pod "eff84f5c-617a-4dcf-8bda-ba203f11f648" (UID: "eff84f5c-617a-4dcf-8bda-ba203f11f648"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:54.938667 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:54.938487 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6017f42-4ca1-4feb-a33a-c2eaf1585d8d-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:54.938667 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:54.938511 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff84f5c-617a-4dcf-8bda-ba203f11f648-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:54.939940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:54.939920 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff84f5c-617a-4dcf-8bda-ba203f11f648-kube-api-access-d4hmj" pod "eff84f5c-617a-4dcf-8bda-ba203f11f648" (UID: "eff84f5c-617a-4dcf-8bda-ba203f11f648"). InnerVolumeSpecName "kube-api-access-d4hmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:54.940017 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:54.939952 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6017f42-4ca1-4feb-a33a-c2eaf1585d8d-kube-api-access-h7xf8" pod "f6017f42-4ca1-4feb-a33a-c2eaf1585d8d" (UID: "f6017f42-4ca1-4feb-a33a-c2eaf1585d8d"). InnerVolumeSpecName "kube-api-access-h7xf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:55.039956 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.039852 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4hmj\" (UniqueName: \"kubernetes.io/projected/eff84f5c-617a-4dcf-8bda-ba203f11f648-kube-api-access-d4hmj\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:55.039956 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.039904 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7xf8\" (UniqueName: \"kubernetes.io/projected/f6017f42-4ca1-4feb-a33a-c2eaf1585d8d-kube-api-access-h7xf8\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:55.318393 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.318303 5111 generic.go:356] "Generic (PLEG): container finished" podID="eff84f5c-617a-4dcf-8bda-ba203f11f648" containerID="7186bc456140056b5b92e4e24c67de4757116df8cebb98d04151e646ecfe08b5" exitCode=0 Nov 03 20:20:55.318393 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.318382 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" Nov 03 20:20:55.318684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.318392 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" event={"ID":"eff84f5c-617a-4dcf-8bda-ba203f11f648","Type":"ContainerDied","Data":"7186bc456140056b5b92e4e24c67de4757116df8cebb98d04151e646ecfe08b5"} Nov 03 20:20:55.318684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.318483 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz" event={"ID":"eff84f5c-617a-4dcf-8bda-ba203f11f648","Type":"ContainerDied","Data":"5456df645417f45c796a2f603344dada1810786e4f47a79607860dea89cda2c7"} Nov 03 20:20:55.318684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.318512 5111 scope.go:122] "RemoveContainer" containerID="7186bc456140056b5b92e4e24c67de4757116df8cebb98d04151e646ecfe08b5" Nov 03 20:20:55.324011 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.323979 5111 generic.go:356] "Generic (PLEG): container finished" podID="f6017f42-4ca1-4feb-a33a-c2eaf1585d8d" containerID="a2c8a83ad7e0d0f5ae42c85e4b22c62f9424cb7b779f58bccbbc4335aa0793fc" exitCode=0 Nov 03 20:20:55.324177 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.324043 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" Nov 03 20:20:55.324177 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.324058 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" event={"ID":"f6017f42-4ca1-4feb-a33a-c2eaf1585d8d","Type":"ContainerDied","Data":"a2c8a83ad7e0d0f5ae42c85e4b22c62f9424cb7b779f58bccbbc4335aa0793fc"} Nov 03 20:20:55.324177 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.324134 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq" event={"ID":"f6017f42-4ca1-4feb-a33a-c2eaf1585d8d","Type":"ContainerDied","Data":"20a36cec00b40d9df3cdb2249223b7ce3ffd6d904e3a87c685d876d2b3945471"} Nov 03 20:20:55.339748 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.339723 5111 scope.go:122] "RemoveContainer" containerID="7186bc456140056b5b92e4e24c67de4757116df8cebb98d04151e646ecfe08b5" Nov 03 20:20:55.340318 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:20:55.340275 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7186bc456140056b5b92e4e24c67de4757116df8cebb98d04151e646ecfe08b5\": container with ID starting with 7186bc456140056b5b92e4e24c67de4757116df8cebb98d04151e646ecfe08b5 not found: ID does not exist" containerID="7186bc456140056b5b92e4e24c67de4757116df8cebb98d04151e646ecfe08b5" Nov 03 20:20:55.340423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.340313 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7186bc456140056b5b92e4e24c67de4757116df8cebb98d04151e646ecfe08b5"} err="failed to get container status \"7186bc456140056b5b92e4e24c67de4757116df8cebb98d04151e646ecfe08b5\": rpc error: code = NotFound desc = could not find container \"7186bc456140056b5b92e4e24c67de4757116df8cebb98d04151e646ecfe08b5\": container with ID starting with 7186bc456140056b5b92e4e24c67de4757116df8cebb98d04151e646ecfe08b5 not found: ID does not exist" Nov 03 20:20:55.340423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.340336 5111 scope.go:122] "RemoveContainer" containerID="a2c8a83ad7e0d0f5ae42c85e4b22c62f9424cb7b779f58bccbbc4335aa0793fc" Nov 03 20:20:55.352905 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.352876 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz"] Nov 03 20:20:55.355117 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.355008 5111 scope.go:122] "RemoveContainer" containerID="a2c8a83ad7e0d0f5ae42c85e4b22c62f9424cb7b779f58bccbbc4335aa0793fc" Nov 03 20:20:55.355429 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:20:55.355405 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c8a83ad7e0d0f5ae42c85e4b22c62f9424cb7b779f58bccbbc4335aa0793fc\": container with ID starting with a2c8a83ad7e0d0f5ae42c85e4b22c62f9424cb7b779f58bccbbc4335aa0793fc not found: ID does not exist" containerID="a2c8a83ad7e0d0f5ae42c85e4b22c62f9424cb7b779f58bccbbc4335aa0793fc" Nov 03 20:20:55.355509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.355439 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c8a83ad7e0d0f5ae42c85e4b22c62f9424cb7b779f58bccbbc4335aa0793fc"} err="failed to get container status \"a2c8a83ad7e0d0f5ae42c85e4b22c62f9424cb7b779f58bccbbc4335aa0793fc\": rpc error: code = NotFound desc = could not find container \"a2c8a83ad7e0d0f5ae42c85e4b22c62f9424cb7b779f58bccbbc4335aa0793fc\": container with ID starting with a2c8a83ad7e0d0f5ae42c85e4b22c62f9424cb7b779f58bccbbc4335aa0793fc not found: ID does not exist" Nov 03 20:20:55.357358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.357335 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-babea4-6464c57675-xsvxz"] Nov 03 20:20:55.364311 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.364277 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq"] Nov 03 20:20:55.368783 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:55.368760 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-babea4-69dd894f84-vwzdq"] Nov 03 20:20:56.185634 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:56.185597 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff84f5c-617a-4dcf-8bda-ba203f11f648" path="/var/lib/kubelet/pods/eff84f5c-617a-4dcf-8bda-ba203f11f648/volumes" Nov 03 20:20:56.186295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:20:56.186272 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6017f42-4ca1-4feb-a33a-c2eaf1585d8d" path="/var/lib/kubelet/pods/f6017f42-4ca1-4feb-a33a-c2eaf1585d8d/volumes" Nov 03 20:21:05.060104 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:21:05.060064 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:21:14.779594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.779551 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl"] Nov 03 20:21:14.780557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.780516 5111 cpu_manager.go:414] "containerMap: removing container" podUID="eff84f5c-617a-4dcf-8bda-ba203f11f648" containerName="payload-pre-processing" Nov 03 20:21:14.780557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.780555 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff84f5c-617a-4dcf-8bda-ba203f11f648" containerName="payload-pre-processing" Nov 03 20:21:14.780733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.780589 5111 cpu_manager.go:414] "containerMap: removing container" podUID="330ae291-dce5-435f-aac6-5cbc4214c273" containerName="istio-proxy" Nov 03 20:21:14.780733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.780596 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330ae291-dce5-435f-aac6-5cbc4214c273" containerName="istio-proxy" Nov 03 20:21:14.780733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.780614 5111 cpu_manager.go:414] "containerMap: removing container" podUID="653d951c-536d-405b-bc11-c9985924b511" containerName="istio-proxy" Nov 03 20:21:14.780733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.780622 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653d951c-536d-405b-bc11-c9985924b511" containerName="istio-proxy" Nov 03 20:21:14.780733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.780652 5111 cpu_manager.go:414] "containerMap: removing container" podUID="f6017f42-4ca1-4feb-a33a-c2eaf1585d8d" containerName="payload-processing" Nov 03 20:21:14.780733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.780659 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6017f42-4ca1-4feb-a33a-c2eaf1585d8d" containerName="payload-processing" Nov 03 20:21:14.781031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.780855 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff84f5c-617a-4dcf-8bda-ba203f11f648" containerName="payload-pre-processing" Nov 03 20:21:14.781031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.780872 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="e140c17f-f261-4a57-938d-7199ddcce0a3" containerName="istio-proxy" Nov 03 20:21:14.781031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.780882 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="330ae291-dce5-435f-aac6-5cbc4214c273" containerName="istio-proxy" Nov 03 20:21:14.781031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.780893 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="653d951c-536d-405b-bc11-c9985924b511" containerName="istio-proxy" Nov 03 20:21:14.781031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.780907 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6017f42-4ca1-4feb-a33a-c2eaf1585d8d" containerName="payload-processing" Nov 03 20:21:14.784921 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.784901 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:14.790202 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.790167 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0a7343-openshift-default-dockercfg-qgfmb\"" Nov 03 20:21:14.798110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.798084 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl"] Nov 03 20:21:14.899819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.899775 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-envoy\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:14.900036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.899852 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istiod-ca-cert\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:14.900036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.899986 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-workload-certs\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:14.900136 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.900058 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-token\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:14.900136 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.900104 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-podinfo\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:14.900251 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.900220 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-data\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:14.900309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.900296 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-credential-socket\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:14.900409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.900392 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqbzn\" (UniqueName: \"kubernetes.io/projected/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-kube-api-access-zqbzn\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:14.900470 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:14.900424 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-workload-socket\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.002072 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.002034 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istiod-ca-cert\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.002285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.002135 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-workload-certs\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.002285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.002180 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-token\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.002285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.002227 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-podinfo\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.002466 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.002299 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-data\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.002466 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.002374 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-credential-socket\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.002604 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.002471 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqbzn\" (UniqueName: \"kubernetes.io/projected/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-kube-api-access-zqbzn\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.002604 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.002498 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-workload-socket\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.002738 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.002622 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-workload-certs\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.002738 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.002727 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-envoy\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.002864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.002754 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-data\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.002864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.002778 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-credential-socket\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.002981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.002875 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istiod-ca-cert\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.002981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.002964 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-workload-socket\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.004889 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.004864 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-envoy\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.005004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.004983 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-podinfo\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.010943 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.010919 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-token\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.012138 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.012112 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqbzn\" (UniqueName: \"kubernetes.io/projected/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-kube-api-access-zqbzn\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.103267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.103177 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:15.395681 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.395632 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl"] Nov 03 20:21:15.397432 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:21:15.397404 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f56c4a1_e91a_4200_ae75_a2f91cc8ac0c.slice/crio-136e2b3a6877d53248f60d2834328f972b8f891b0874f1e1d08fa2244898da61 WatchSource:0}: Error finding container 136e2b3a6877d53248f60d2834328f972b8f891b0874f1e1d08fa2244898da61: Status 404 returned error can't find the container with id 136e2b3a6877d53248f60d2834328f972b8f891b0874f1e1d08fa2244898da61 Nov 03 20:21:15.401333 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.401290 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:15.401454 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.401378 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:15.401454 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.401410 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:15.679440 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.679340 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" event={"ID":"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c","Type":"ContainerStarted","Data":"6ed9c0d6de305e562e5855fbee5080107934d155639e2801bde5743c7c659ef5"} Nov 03 20:21:15.679440 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.679411 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" event={"ID":"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c","Type":"ContainerStarted","Data":"136e2b3a6877d53248f60d2834328f972b8f891b0874f1e1d08fa2244898da61"} Nov 03 20:21:15.707077 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:15.707024 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" podStartSLOduration=1.7070091120000002 podStartE2EDuration="1.707009112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:15.701592907 +0000 UTC m=+2505.663971345" watchObservedRunningTime="2026-11-03 20:21:15.707009112 +0000 UTC m=+2505.669387550" Nov 03 20:21:16.103930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:16.103831 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:16.105463 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:16.105436 5111 patch_prober.go:29] interesting pod/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 20:21:16.105632 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:16.105484 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" podUID="6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 20:21:17.104749 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:17.104001 5111 patch_prober.go:29] interesting pod/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 20:21:17.104749 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:17.104081 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" podUID="6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 20:21:17.476066 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:17.476011 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8"] Nov 03 20:21:17.654264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:17.654229 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8"] Nov 03 20:21:17.654635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:17.654605 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" Nov 03 20:21:17.728309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:17.728222 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-a8e19e-dockercfg-m8prr\"" Nov 03 20:21:17.815287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:17.794027 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m9mc\" (UniqueName: \"kubernetes.io/projected/9792b23b-9450-44e3-bb6c-c90a9d5411fa-kube-api-access-7m9mc\") pod \"payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8\" (UID: \"9792b23b-9450-44e3-bb6c-c90a9d5411fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" Nov 03 20:21:17.816135 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:17.816079 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9792b23b-9450-44e3-bb6c-c90a9d5411fa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8\" (UID: \"9792b23b-9450-44e3-bb6c-c90a9d5411fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" Nov 03 20:21:17.816546 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:17.794323 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-a8e19e\"" Nov 03 20:21:17.917769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:17.917722 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9792b23b-9450-44e3-bb6c-c90a9d5411fa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8\" (UID: \"9792b23b-9450-44e3-bb6c-c90a9d5411fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" Nov 03 20:21:17.917982 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:17.917814 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9mc\" (UniqueName: \"kubernetes.io/projected/9792b23b-9450-44e3-bb6c-c90a9d5411fa-kube-api-access-7m9mc\") pod \"payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8\" (UID: \"9792b23b-9450-44e3-bb6c-c90a9d5411fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" Nov 03 20:21:17.918622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:17.918585 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9792b23b-9450-44e3-bb6c-c90a9d5411fa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8\" (UID: \"9792b23b-9450-44e3-bb6c-c90a9d5411fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" Nov 03 20:21:17.926389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:17.926364 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m9mc\" (UniqueName: \"kubernetes.io/projected/9792b23b-9450-44e3-bb6c-c90a9d5411fa-kube-api-access-7m9mc\") pod \"payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8\" (UID: \"9792b23b-9450-44e3-bb6c-c90a9d5411fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" Nov 03 20:21:18.105306 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:18.105205 5111 patch_prober.go:29] interesting pod/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 20:21:18.105306 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:18.105269 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" podUID="6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 20:21:18.139560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:18.137501 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" Nov 03 20:21:18.526279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:18.526246 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8"] Nov 03 20:21:18.526471 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:21:18.526444 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9792b23b_9450_44e3_bb6c_c90a9d5411fa.slice/crio-9a278df1da708879a0cde4068fe0a32532be57e3c7b90e441f8ea7fceec338e3 WatchSource:0}: Error finding container 9a278df1da708879a0cde4068fe0a32532be57e3c7b90e441f8ea7fceec338e3: Status 404 returned error can't find the container with id 9a278df1da708879a0cde4068fe0a32532be57e3c7b90e441f8ea7fceec338e3 Nov 03 20:21:18.871614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:18.867143 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" event={"ID":"9792b23b-9450-44e3-bb6c-c90a9d5411fa","Type":"ContainerStarted","Data":"3064cf0c4d5b2cb92d0c5dd509ea4ca959e99e1d97f06afb22784614348b697b"} Nov 03 20:21:18.871614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:18.867251 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" event={"ID":"9792b23b-9450-44e3-bb6c-c90a9d5411fa","Type":"ContainerStarted","Data":"9a278df1da708879a0cde4068fe0a32532be57e3c7b90e441f8ea7fceec338e3"} Nov 03 20:21:18.888557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:18.876612 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" Nov 03 20:21:18.909970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:18.904494 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" podStartSLOduration=1.904473973 podStartE2EDuration="1.904473973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:18.901931896 +0000 UTC m=+2508.864310335" watchObservedRunningTime="2026-11-03 20:21:18.904473973 +0000 UTC m=+2508.866852410" Nov 03 20:21:19.107561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.106919 5111 patch_prober.go:29] interesting pod/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 20:21:19.107561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.106982 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" podUID="6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 20:21:19.191963 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.191877 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57"] Nov 03 20:21:19.196883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.196860 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.204372 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.204343 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-ee2323-openshift-default-dockercfg-krfdd\"" Nov 03 20:21:19.213395 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.211746 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57"] Nov 03 20:21:19.277301 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.277111 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7608cd01-af9c-4819-9016-6c010e0615ec-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.277301 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.277254 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.277301 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.277293 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.277697 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.277318 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qmfd\" (UniqueName: \"kubernetes.io/projected/7608cd01-af9c-4819-9016-6c010e0615ec-kube-api-access-8qmfd\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.277697 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.277514 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7608cd01-af9c-4819-9016-6c010e0615ec-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.277697 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.277653 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.277874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.277714 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.277874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.277755 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7608cd01-af9c-4819-9016-6c010e0615ec-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.277874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.277850 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.380486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.380260 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7608cd01-af9c-4819-9016-6c010e0615ec-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.380486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.380325 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.380486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.380390 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.380486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.380436 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7608cd01-af9c-4819-9016-6c010e0615ec-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.388831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.380521 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.388831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.380600 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7608cd01-af9c-4819-9016-6c010e0615ec-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.388831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.380679 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.388831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.380717 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.388831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.380748 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qmfd\" (UniqueName: \"kubernetes.io/projected/7608cd01-af9c-4819-9016-6c010e0615ec-kube-api-access-8qmfd\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.388831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.381235 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.388831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.381368 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.388831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.381959 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.388831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.384191 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7608cd01-af9c-4819-9016-6c010e0615ec-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.401984 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.394758 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.405361 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.405262 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.470686 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.445228 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7608cd01-af9c-4819-9016-6c010e0615ec-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.470686 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.456204 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7608cd01-af9c-4819-9016-6c010e0615ec-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.494874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.494325 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qmfd\" (UniqueName: \"kubernetes.io/projected/7608cd01-af9c-4819-9016-6c010e0615ec-kube-api-access-8qmfd\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.519029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.518989 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:19.591684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.591649 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8"] Nov 03 20:21:19.598584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.598485 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:19.605678 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.605465 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8"] Nov 03 20:21:19.609519 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.609489 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b55105-openshift-default-dockercfg-b42xt\"" Nov 03 20:21:19.685474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.685440 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-workload-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:19.685613 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.685490 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6gmh\" (UniqueName: \"kubernetes.io/projected/2815d45f-e02c-4178-9f95-0375fda5f11c-kube-api-access-n6gmh\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:19.685690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.685527 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-envoy\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:19.685821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.685801 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-workload-certs\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:19.685859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.685849 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2815d45f-e02c-4178-9f95-0375fda5f11c-istiod-ca-cert\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:19.685897 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.685882 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-podinfo\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:19.685935 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.685926 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-data\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:19.685986 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.685970 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-credential-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:19.686089 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:19.686073 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-token\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.046573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.036409 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-token\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.046573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.036503 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-workload-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.097731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.094643 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-workload-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.097731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.036527 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6gmh\" (UniqueName: \"kubernetes.io/projected/2815d45f-e02c-4178-9f95-0375fda5f11c-kube-api-access-n6gmh\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.097731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.094765 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-envoy\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.097731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.095091 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-workload-certs\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.097731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.095173 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2815d45f-e02c-4178-9f95-0375fda5f11c-istiod-ca-cert\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.097731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.095207 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-podinfo\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.097731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.095259 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-data\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.097731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.095311 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-credential-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.097731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.095842 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-credential-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.112171 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.112133 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2815d45f-e02c-4178-9f95-0375fda5f11c-istiod-ca-cert\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.112768 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.112433 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-workload-certs\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.116335 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.116302 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-data\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.116601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.116558 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-podinfo\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.124238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.121218 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57"] Nov 03 20:21:20.125063 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.125016 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6gmh\" (UniqueName: \"kubernetes.io/projected/2815d45f-e02c-4178-9f95-0375fda5f11c-kube-api-access-n6gmh\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.125595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.125474 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-envoy\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.126242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.126192 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-token\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.144433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.133691 5111 patch_prober.go:29] interesting pod/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 20:21:20.144433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.133739 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" podUID="6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 20:21:20.144433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.139796 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:20.144433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.139890 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:20.144433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.139929 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:20.278791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.273064 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:20.613019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.612963 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:20.613165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.613133 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:20.613247 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.613235 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:20.620260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:20.620063 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8"] Nov 03 20:21:21.104299 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:21.104254 5111 patch_prober.go:29] interesting pod/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 20:21:21.104500 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:21.104316 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" podUID="6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 20:21:21.142762 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:21.142713 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" event={"ID":"2815d45f-e02c-4178-9f95-0375fda5f11c","Type":"ContainerStarted","Data":"f237b247cefe086452423efa17b16ea60f036937087d5c6d50da4f16ebb0448b"} Nov 03 20:21:21.143228 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:21.142778 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" event={"ID":"2815d45f-e02c-4178-9f95-0375fda5f11c","Type":"ContainerStarted","Data":"826f0283964bc89f6089a897d6fef0a4248efbfa93095eac2661ab7c02ca07ad"} Nov 03 20:21:21.149559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:21.144629 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" event={"ID":"7608cd01-af9c-4819-9016-6c010e0615ec","Type":"ContainerStarted","Data":"8025b778a3669dc573ea6a44cb67407245fc582d27f0148a58c14a7f7f174a11"} Nov 03 20:21:21.149559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:21.144685 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" event={"ID":"7608cd01-af9c-4819-9016-6c010e0615ec","Type":"ContainerStarted","Data":"971f3cf1cdf7b10d69531a0cdab496bd956cbf420d9002b53ebc8839dffde7a8"} Nov 03 20:21:21.176364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:21.176297 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" podStartSLOduration=2.176279156 podStartE2EDuration="2.176279156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:21.174168689 +0000 UTC m=+2511.136547133" watchObservedRunningTime="2026-11-03 20:21:21.176279156 +0000 UTC m=+2511.138657596" Nov 03 20:21:21.199482 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:21.195159 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" podStartSLOduration=2.195140016 podStartE2EDuration="2.195140016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:21.194588442 +0000 UTC m=+2511.156966881" watchObservedRunningTime="2026-11-03 20:21:21.195140016 +0000 UTC m=+2511.157518457" Nov 03 20:21:21.275132 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:21.275088 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:21.279050 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:21.279008 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 20:21:21.279341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:21.279312 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 20:21:21.522352 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:21.521662 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:21.522352 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:21.521988 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 20:21:21.522352 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:21.522125 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" podUID="7608cd01-af9c-4819-9016-6c010e0615ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 20:21:22.103718 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:22.103683 5111 patch_prober.go:29] interesting pod/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 20:21:22.103883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:22.103738 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" podUID="6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 20:21:22.282155 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:22.282118 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:22.520385 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:22.520344 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 20:21:22.520635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:22.520406 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" podUID="7608cd01-af9c-4819-9016-6c010e0615ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 20:21:23.109109 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:23.109075 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:23.109568 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:23.109552 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:23.110348 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:23.110325 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:21:23.216763 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:23.216732 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:23.218021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:23.217993 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:21:23.519714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:23.519680 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 20:21:23.520160 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:23.519735 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" podUID="7608cd01-af9c-4819-9016-6c010e0615ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 20:21:24.519427 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:24.519388 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 20:21:24.519646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:24.519454 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" podUID="7608cd01-af9c-4819-9016-6c010e0615ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 20:21:25.019560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:25.005096 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb"] Nov 03 20:21:25.059769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:25.051336 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" Nov 03 20:21:25.116559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:25.100909 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-b55105\"" Nov 03 20:21:25.116559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:25.101071 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-b55105-dockercfg-9m8kd\"" Nov 03 20:21:25.248571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:25.237171 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8dwb\" (UniqueName: \"kubernetes.io/projected/6f695bca-daf7-4b45-baab-e4b6a5c5adbc-kube-api-access-f8dwb\") pod \"payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb\" (UID: \"6f695bca-daf7-4b45-baab-e4b6a5c5adbc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" Nov 03 20:21:25.248571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:25.237319 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f695bca-daf7-4b45-baab-e4b6a5c5adbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb\" (UID: \"6f695bca-daf7-4b45-baab-e4b6a5c5adbc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" Nov 03 20:21:25.248571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:25.244718 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb"] Nov 03 20:21:25.406292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:25.404591 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f695bca-daf7-4b45-baab-e4b6a5c5adbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb\" (UID: \"6f695bca-daf7-4b45-baab-e4b6a5c5adbc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" Nov 03 20:21:25.406292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:25.405133 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8dwb\" (UniqueName: \"kubernetes.io/projected/6f695bca-daf7-4b45-baab-e4b6a5c5adbc-kube-api-access-f8dwb\") pod \"payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb\" (UID: \"6f695bca-daf7-4b45-baab-e4b6a5c5adbc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" Nov 03 20:21:25.415303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:25.415269 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f695bca-daf7-4b45-baab-e4b6a5c5adbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb\" (UID: \"6f695bca-daf7-4b45-baab-e4b6a5c5adbc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" Nov 03 20:21:25.516647 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:25.497319 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8dwb\" (UniqueName: \"kubernetes.io/projected/6f695bca-daf7-4b45-baab-e4b6a5c5adbc-kube-api-access-f8dwb\") pod \"payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb\" (UID: \"6f695bca-daf7-4b45-baab-e4b6a5c5adbc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" Nov 03 20:21:25.576618 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:25.555505 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 20:21:25.576618 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:25.555581 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" podUID="7608cd01-af9c-4819-9016-6c010e0615ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 20:21:25.685556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:25.684694 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" Nov 03 20:21:26.121795 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:26.121760 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb"] Nov 03 20:21:26.134007 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:21:26.129673 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f695bca_daf7_4b45_baab_e4b6a5c5adbc.slice/crio-394bcc5a6d2f3c4098c4a0186dc8d092c92f4b2232ba1c8ff5f471dbf5037a0d WatchSource:0}: Error finding container 394bcc5a6d2f3c4098c4a0186dc8d092c92f4b2232ba1c8ff5f471dbf5037a0d: Status 404 returned error can't find the container with id 394bcc5a6d2f3c4098c4a0186dc8d092c92f4b2232ba1c8ff5f471dbf5037a0d Nov 03 20:21:26.588938 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:26.583004 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 20:21:26.588938 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:26.583064 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" podUID="7608cd01-af9c-4819-9016-6c010e0615ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 20:21:26.740664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:26.740494 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" event={"ID":"6f695bca-daf7-4b45-baab-e4b6a5c5adbc","Type":"ContainerStarted","Data":"3a61961c9210b987d9c50a2a3f7fe7304f25a45f10ac5626e320ed274fe22257"} Nov 03 20:21:26.740664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:26.740619 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" event={"ID":"6f695bca-daf7-4b45-baab-e4b6a5c5adbc","Type":"ContainerStarted","Data":"394bcc5a6d2f3c4098c4a0186dc8d092c92f4b2232ba1c8ff5f471dbf5037a0d"} Nov 03 20:21:26.741332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:26.741005 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" Nov 03 20:21:26.798140 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:26.798070 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" podStartSLOduration=2.7980459079999997 podStartE2EDuration="2.798045908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:26.771966116 +0000 UTC m=+2516.734344556" watchObservedRunningTime="2026-11-03 20:21:26.798045908 +0000 UTC m=+2516.760424347" Nov 03 20:21:27.261554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:27.255690 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:27.283558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:27.281650 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:27.410553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:27.409733 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:27.541604 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:27.534718 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 20:21:27.541604 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:27.534778 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" podUID="7608cd01-af9c-4819-9016-6c010e0615ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 20:21:27.565115 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:27.558671 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:28.010573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:28.003803 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:28.054567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:28.046461 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:28.197557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:28.189864 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:28.273612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:28.259635 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:28.273612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:28.260218 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:21:28.435292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:28.435253 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:21:28.528618 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:28.528137 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 20:21:28.528618 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:28.528207 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" podUID="7608cd01-af9c-4819-9016-6c010e0615ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 20:21:29.543888 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:29.543843 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:29.564311 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:29.564273 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:29.584842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:29.584773 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:21:30.135739 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:30.135712 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" Nov 03 20:21:34.203369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.203337 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr"] Nov 03 20:21:34.212612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.211621 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.234462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.234407 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr"] Nov 03 20:21:34.250490 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.250451 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx"] Nov 03 20:21:34.281403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.280477 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx"] Nov 03 20:21:34.281403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.280617 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.390833 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.390795 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b31fd8-6624-466d-a5cf-2c84b5305b29-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.391036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.390854 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.391036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.390961 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.391036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.390993 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.391036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.391020 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.391208 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.391157 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcm5q\" (UniqueName: \"kubernetes.io/projected/5d4d92cc-f4fa-421c-8b95-89838debd9b4-kube-api-access-mcm5q\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.391251 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.391204 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.391336 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.391320 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.391437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.391422 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.391507 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.391458 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.391639 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.391591 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.391698 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.391642 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxlz5\" (UniqueName: \"kubernetes.io/projected/95b31fd8-6624-466d-a5cf-2c84b5305b29-kube-api-access-kxlz5\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.391698 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.391681 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.391779 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.391713 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.391779 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.391753 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.391845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.391788 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.391845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.391814 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.391845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.391835 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.493427 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.493319 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.493427 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.493395 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.493725 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.493439 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxlz5\" (UniqueName: \"kubernetes.io/projected/95b31fd8-6624-466d-a5cf-2c84b5305b29-kube-api-access-kxlz5\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.493725 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.493474 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.493725 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.493508 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.493725 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.493576 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.493725 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.493620 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.493725 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.493659 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.493725 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.493692 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.494148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.493804 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b31fd8-6624-466d-a5cf-2c84b5305b29-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.494148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.493826 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.494148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.493843 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.494148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.493872 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.494148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.493895 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.494148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.493932 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.494148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.494007 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcm5q\" (UniqueName: \"kubernetes.io/projected/5d4d92cc-f4fa-421c-8b95-89838debd9b4-kube-api-access-mcm5q\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.494148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.494055 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.494148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.494139 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.494736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.494186 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.494736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.494252 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.494736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.494254 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.494736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.494466 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.494736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.494601 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.495025 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.494942 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.495457 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.495172 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.495457 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.495402 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.495739 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.495713 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.495900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.495867 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b31fd8-6624-466d-a5cf-2c84b5305b29-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.497739 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.497526 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.498123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.498091 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.498240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.498216 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.498492 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.498474 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.505268 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.505243 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.505637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.505609 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.507149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.507122 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcm5q\" (UniqueName: \"kubernetes.io/projected/5d4d92cc-f4fa-421c-8b95-89838debd9b4-kube-api-access-mcm5q\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.511766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.511719 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxlz5\" (UniqueName: \"kubernetes.io/projected/95b31fd8-6624-466d-a5cf-2c84b5305b29-kube-api-access-kxlz5\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:34.590333 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.590296 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:34.607142 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:34.607107 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:35.091810 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:21:35.091722 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:21:35.166625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:35.166577 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:35.166760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:35.166679 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:35.166760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:35.166745 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:35.167011 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:35.166931 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr"] Nov 03 20:21:35.167130 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:21:35.167029 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b31fd8_6624_466d_a5cf_2c84b5305b29.slice/crio-cafb9029de22c718d6d509582cbd1e41a1ff44ad1bde495d55cff39287d5367f WatchSource:0}: Error finding container cafb9029de22c718d6d509582cbd1e41a1ff44ad1bde495d55cff39287d5367f: Status 404 returned error can't find the container with id cafb9029de22c718d6d509582cbd1e41a1ff44ad1bde495d55cff39287d5367f Nov 03 20:21:35.171555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:35.171499 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx"] Nov 03 20:21:35.173013 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:35.172980 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:35.173099 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:35.173057 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:35.173156 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:35.173096 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:36.270201 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:36.270161 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" event={"ID":"5d4d92cc-f4fa-421c-8b95-89838debd9b4","Type":"ContainerStarted","Data":"e47df2ee3f51faaea58722cc8fe986a244c67c82605ea24fa0cf50399fbcda5d"} Nov 03 20:21:36.270933 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:36.270879 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" event={"ID":"5d4d92cc-f4fa-421c-8b95-89838debd9b4","Type":"ContainerStarted","Data":"f441a2f061da2eedc3ca6100fbf4d2fd740a34deade6b0531bd541c2a4cb3464"} Nov 03 20:21:36.271105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:36.271085 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" event={"ID":"95b31fd8-6624-466d-a5cf-2c84b5305b29","Type":"ContainerStarted","Data":"87087700fc59440bbde04edafa88932f94d7e8716e085e723f818a99710bcb45"} Nov 03 20:21:36.271228 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:36.271215 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" event={"ID":"95b31fd8-6624-466d-a5cf-2c84b5305b29","Type":"ContainerStarted","Data":"cafb9029de22c718d6d509582cbd1e41a1ff44ad1bde495d55cff39287d5367f"} Nov 03 20:21:36.276327 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:36.276261 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" podStartSLOduration=2.276243614 podStartE2EDuration="2.276243614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:36.272787042 +0000 UTC m=+2526.235165558" watchObservedRunningTime="2026-11-03 20:21:36.276243614 +0000 UTC m=+2526.238622052" Nov 03 20:21:36.302432 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:36.302367 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" podStartSLOduration=2.302347879 podStartE2EDuration="2.302347879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:36.301711577 +0000 UTC m=+2526.264090015" watchObservedRunningTime="2026-11-03 20:21:36.302347879 +0000 UTC m=+2526.264726319" Nov 03 20:21:36.604966 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:36.604817 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:36.608351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:36.608266 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:36.616371 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:36.615750 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 20:21:36.616371 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:36.615809 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" podUID="95b31fd8-6624-466d-a5cf-2c84b5305b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 20:21:36.616371 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:36.616111 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 20:21:36.616371 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:36.616141 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" podUID="5d4d92cc-f4fa-421c-8b95-89838debd9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 20:21:37.594607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:37.594050 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 20:21:37.594607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:37.594119 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" podUID="5d4d92cc-f4fa-421c-8b95-89838debd9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 20:21:37.608804 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:37.608757 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 20:21:37.609046 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:37.608819 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" podUID="95b31fd8-6624-466d-a5cf-2c84b5305b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 20:21:37.950031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:37.949994 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" Nov 03 20:21:38.604719 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:38.594262 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 20:21:38.604719 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:38.594320 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" podUID="5d4d92cc-f4fa-421c-8b95-89838debd9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 20:21:38.617185 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:38.608212 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 20:21:38.617185 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:38.608295 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" podUID="95b31fd8-6624-466d-a5cf-2c84b5305b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 20:21:39.603556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:39.602639 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 20:21:39.603556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:39.602726 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" podUID="5d4d92cc-f4fa-421c-8b95-89838debd9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 20:21:39.608571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:39.608504 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 20:21:39.608980 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:39.608573 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" podUID="95b31fd8-6624-466d-a5cf-2c84b5305b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 20:21:40.609207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:40.604817 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 20:21:40.609207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:40.604888 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" podUID="5d4d92cc-f4fa-421c-8b95-89838debd9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 20:21:40.609207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:40.607998 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 20:21:40.609207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:40.608061 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" podUID="95b31fd8-6624-466d-a5cf-2c84b5305b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 20:21:41.597999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:41.597956 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 20:21:41.598314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:41.598282 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" podUID="5d4d92cc-f4fa-421c-8b95-89838debd9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 20:21:41.620870 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:41.620520 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 20:21:41.620870 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:41.620596 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" podUID="95b31fd8-6624-466d-a5cf-2c84b5305b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 20:21:42.615636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:42.608677 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 20:21:42.615636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:42.608746 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" podUID="95b31fd8-6624-466d-a5cf-2c84b5305b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 20:21:42.779829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:42.779769 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:42.780468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:42.780417 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:42.974552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:42.973803 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:42.977562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:42.975829 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:42.977562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:42.976797 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:21:43.614633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:43.614593 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:43.615088 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:43.615057 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:43.616112 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:43.616090 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:21:48.025732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:48.025680 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:48.026257 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:48.025764 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" podUID="5478802b-2542-47ee-919d-5a8ad79a5e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:49.250907 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.250868 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx"] Nov 03 20:21:49.294379 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.294342 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx"] Nov 03 20:21:49.294566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.294468 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.395180 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.395140 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.395376 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.395214 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.395423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.395384 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.395647 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.395627 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqxzd\" (UniqueName: \"kubernetes.io/projected/45038f2d-5a7b-4f31-92f3-147292e1c517-kube-api-access-cqxzd\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.395716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.395700 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.395758 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.395731 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.395795 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.395778 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.395835 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.395814 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45038f2d-5a7b-4f31-92f3-147292e1c517-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.396029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.396001 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.497556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.497490 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqxzd\" (UniqueName: \"kubernetes.io/projected/45038f2d-5a7b-4f31-92f3-147292e1c517-kube-api-access-cqxzd\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.497769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.497580 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.497769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.497600 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.497769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.497644 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.497769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.497661 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45038f2d-5a7b-4f31-92f3-147292e1c517-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.498011 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.497780 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.498011 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.497845 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.498011 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.497915 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.498011 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.497995 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.498275 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.498080 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.498275 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.498126 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.498557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.498507 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.498765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.498598 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45038f2d-5a7b-4f31-92f3-147292e1c517-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.511835 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.498770 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.530123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.529247 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.532804 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.532777 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqxzd\" (UniqueName: \"kubernetes.io/projected/45038f2d-5a7b-4f31-92f3-147292e1c517-kube-api-access-cqxzd\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.533860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.533826 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.542254 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.542226 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:49.644900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:49.644861 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:50.041897 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:21:50.041843 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45038f2d_5a7b_4f31_92f3_147292e1c517.slice/crio-db1746b791feb57b0bde4ad68e2fe69ac6b4c80626acda5e90bb8fbd2c9cf330 WatchSource:0}: Error finding container db1746b791feb57b0bde4ad68e2fe69ac6b4c80626acda5e90bb8fbd2c9cf330: Status 404 returned error can't find the container with id db1746b791feb57b0bde4ad68e2fe69ac6b4c80626acda5e90bb8fbd2c9cf330 Nov 03 20:21:50.047380 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:50.047338 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:50.047484 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:50.047435 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:50.047527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:50.047478 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:50.051759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:50.051732 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx"] Nov 03 20:21:50.557346 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:50.557301 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" event={"ID":"45038f2d-5a7b-4f31-92f3-147292e1c517","Type":"ContainerStarted","Data":"5fc029674e09a18493b9cfb5cda4b9322291acf45ce1dfcb5eb662eb7924a57c"} Nov 03 20:21:50.557815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:50.557380 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" event={"ID":"45038f2d-5a7b-4f31-92f3-147292e1c517","Type":"ContainerStarted","Data":"db1746b791feb57b0bde4ad68e2fe69ac6b4c80626acda5e90bb8fbd2c9cf330"} Nov 03 20:21:50.586761 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:50.586695 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" podStartSLOduration=1.5866733929999999 podStartE2EDuration="1.586673393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:50.577683932 +0000 UTC m=+2540.540062367" watchObservedRunningTime="2026-11-03 20:21:50.586673393 +0000 UTC m=+2540.549051836" Nov 03 20:21:50.652896 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:50.652854 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:51.671556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:51.670927 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 20:21:51.671556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:51.670999 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" podUID="45038f2d-5a7b-4f31-92f3-147292e1c517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 20:21:52.667558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:52.666356 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 20:21:52.667558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:52.666423 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" podUID="45038f2d-5a7b-4f31-92f3-147292e1c517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 20:21:53.690959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:53.687733 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 20:21:53.690959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:53.687795 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" podUID="45038f2d-5a7b-4f31-92f3-147292e1c517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 20:21:54.293143 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.293099 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl"] Nov 03 20:21:54.322514 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.322474 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl"] Nov 03 20:21:54.322766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.322631 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.407640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.407575 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.407862 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.407790 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.407862 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.407840 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.407997 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.407872 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.407997 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.407938 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.408113 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.408004 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhmwk\" (UniqueName: \"kubernetes.io/projected/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-kube-api-access-hhmwk\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.408113 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.408025 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.408113 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.408062 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.408113 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.408081 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.510126 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.510070 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.510243 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.510289 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.510333 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.510409 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.510504 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhmwk\" (UniqueName: \"kubernetes.io/projected/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-kube-api-access-hhmwk\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.510552 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.510616 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.510666 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.511148 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.511473 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.513688 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.513977 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.514278 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.514623 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.515480 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.519925 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhmwk\" (UniqueName: \"kubernetes.io/projected/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-kube-api-access-hhmwk\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.519941 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.659133 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.659215 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" podUID="45038f2d-5a7b-4f31-92f3-147292e1c517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 20:21:54.908886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:54.687950 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:55.038210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:55.038175 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl"] Nov 03 20:21:55.039896 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:21:55.039673 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47c1ee1_ba29_43e7_a737_5f2a6a8fa56f.slice/crio-4ec2250380d0a80fbbaca8e10e07fbb94f9fdcdaa7c03575d93520adfd82e204 WatchSource:0}: Error finding container 4ec2250380d0a80fbbaca8e10e07fbb94f9fdcdaa7c03575d93520adfd82e204: Status 404 returned error can't find the container with id 4ec2250380d0a80fbbaca8e10e07fbb94f9fdcdaa7c03575d93520adfd82e204 Nov 03 20:21:55.043269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:55.043234 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:55.044474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:55.043316 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:55.044474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:55.043347 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:55.659554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:55.659009 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 20:21:55.659554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:55.659071 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" podUID="45038f2d-5a7b-4f31-92f3-147292e1c517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 20:21:55.690503 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:55.690472 5111 generic.go:356] "Generic (PLEG): container finished" podID="5478802b-2542-47ee-919d-5a8ad79a5e6a" containerID="041b235c029964f198d7c3e2af1e79a8f57e3c3b45d4528370b72a2cac25709b" exitCode=0 Nov 03 20:21:55.690728 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:55.690578 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" event={"ID":"5478802b-2542-47ee-919d-5a8ad79a5e6a","Type":"ContainerDied","Data":"041b235c029964f198d7c3e2af1e79a8f57e3c3b45d4528370b72a2cac25709b"} Nov 03 20:21:55.691236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:55.691215 5111 scope.go:122] "RemoveContainer" containerID="041b235c029964f198d7c3e2af1e79a8f57e3c3b45d4528370b72a2cac25709b" Nov 03 20:21:55.692977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:55.692950 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" event={"ID":"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f","Type":"ContainerStarted","Data":"8cbe909a2f6df9ace58ef88b130ed5d0c0d84709df84c735a7ec87bf0a7bf177"} Nov 03 20:21:55.693079 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:55.692998 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" event={"ID":"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f","Type":"ContainerStarted","Data":"4ec2250380d0a80fbbaca8e10e07fbb94f9fdcdaa7c03575d93520adfd82e204"} Nov 03 20:21:55.693634 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:55.693605 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:55.693713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:55.693691 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:55.693787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:55.693752 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:55.733109 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:55.733045 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" podStartSLOduration=1.733026308 podStartE2EDuration="1.733026308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:55.729741188 +0000 UTC m=+2545.692119626" watchObservedRunningTime="2026-11-03 20:21:55.733026308 +0000 UTC m=+2545.695404746" Nov 03 20:21:56.665562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:56.655009 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 20:21:56.665562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:56.655076 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" podUID="45038f2d-5a7b-4f31-92f3-147292e1c517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 20:21:56.711227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:56.690138 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:21:56.711227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:56.690246 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 20:21:56.711227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:56.690283 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" podUID="f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 20:21:56.711227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:56.705236 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" event={"ID":"5478802b-2542-47ee-919d-5a8ad79a5e6a","Type":"ContainerStarted","Data":"44fe7c39f39f7049fe7af14e26eaf9541523be6e3ca1a9b143d96a7c21035236"} Nov 03 20:21:56.832287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:56.832247 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": read tcp 10.132.0.2:55408->10.132.0.179:15021: read: connection reset by peer" start-of-body= Nov 03 20:21:56.832477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:56.832321 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": read tcp 10.132.0.2:55408->10.132.0.179:15021: read: connection reset by peer" Nov 03 20:21:57.533560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:57.533269 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:21:57.552554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:57.539473 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Nov 03 20:21:57.552554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:57.539555 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" podUID="5478802b-2542-47ee-919d-5a8ad79a5e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Nov 03 20:21:57.683557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:57.682891 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:57.696848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:57.684733 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:57.696848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:57.687965 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:21:57.696848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:57.690312 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 20:21:57.696848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:57.690359 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" podUID="f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 20:21:57.715979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:57.715948 5111 generic.go:356] "Generic (PLEG): container finished" podID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerID="dfa0cc2e93fd859038ce9bd941bc694d22671bbd001e35bf883153a042ff2b19" exitCode=0 Nov 03 20:21:57.716159 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:57.716022 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" event={"ID":"027fe01b-c13b-4f05-babf-c440793dfa7d","Type":"ContainerDied","Data":"dfa0cc2e93fd859038ce9bd941bc694d22671bbd001e35bf883153a042ff2b19"} Nov 03 20:21:57.716597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:57.716570 5111 scope.go:122] "RemoveContainer" containerID="dfa0cc2e93fd859038ce9bd941bc694d22671bbd001e35bf883153a042ff2b19" Nov 03 20:21:57.718348 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:57.718319 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:57.718430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:57.718385 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:57.718430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:57.718416 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:21:58.540023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:58.539928 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Nov 03 20:21:58.540231 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:58.540052 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" podUID="5478802b-2542-47ee-919d-5a8ad79a5e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Nov 03 20:21:58.698978 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:58.698940 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 20:21:58.699646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:58.699593 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" podUID="f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 20:21:58.728651 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:58.728613 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" event={"ID":"027fe01b-c13b-4f05-babf-c440793dfa7d","Type":"ContainerStarted","Data":"e1d8e1211010cf0fc71792f2cd58c8a5a95de6b2311ec1d8e5dbaa16424ad724"} Nov 03 20:21:58.768283 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:58.768244 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:21:58.768677 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:58.768586 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:21:58.768855 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:58.768832 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 20:21:59.529146 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:59.529108 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Nov 03 20:21:59.529451 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:59.529420 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" podUID="5478802b-2542-47ee-919d-5a8ad79a5e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Nov 03 20:21:59.917002 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:59.916960 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:21:59.917656 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:59.917618 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 20:21:59.929698 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:59.927137 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 20:21:59.929698 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:21:59.927191 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" podUID="f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 20:22:00.529484 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:00.529444 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Nov 03 20:22:00.529830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:00.529796 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" podUID="5478802b-2542-47ee-919d-5a8ad79a5e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Nov 03 20:22:00.689444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:00.688973 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 20:22:00.689444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:00.689041 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" podUID="f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 20:22:00.764149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:00.764104 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:22:00.764451 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:00.764419 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 20:22:01.529277 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:01.529238 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Nov 03 20:22:01.529850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:01.529641 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" podUID="5478802b-2542-47ee-919d-5a8ad79a5e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Nov 03 20:22:01.689788 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:01.689745 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 20:22:01.690093 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:01.690063 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" podUID="f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 20:22:01.766556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:01.765963 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:22:01.766556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:01.766245 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 20:22:01.987786 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:01.987749 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:22:02.313255 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:02.313157 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c"] Nov 03 20:22:02.313477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:02.313448 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" podUID="0f667fce-363f-476e-8601-afd78a7f3121" containerName="istio-proxy" containerID="cri-o://fe9bef312215087c89b4e4f4e014ecc2813ca74b37f58c5b67a1d092cacf5aa4" gracePeriod=30 Nov 03 20:22:02.528662 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:02.528626 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-p994c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:22:02.528916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:02.528892 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" podUID="0f667fce-363f-476e-8601-afd78a7f3121" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:22:02.529886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:02.529861 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Nov 03 20:22:02.530390 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:02.530367 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" podUID="5478802b-2542-47ee-919d-5a8ad79a5e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Nov 03 20:22:02.766736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:02.766698 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:22:02.767050 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:02.767021 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 20:22:03.178557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:03.167039 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:22:03.178557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:03.169030 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:22:03.185552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:03.181152 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:22:03.586509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:03.586409 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:22:03.588985 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:03.588960 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:22:03.756029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:03.755972 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:22:03.756427 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:03.756375 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 20:22:04.769558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:04.768905 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:22:04.769558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:04.768969 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 20:22:05.128456 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:22:05.128420 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:22:05.756848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:05.756809 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:22:05.757164 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:05.757128 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 20:22:06.760784 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:06.760737 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:22:06.761263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:06.761240 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:22:06.762003 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:06.761981 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:22:07.630039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.630011 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 20:22:07.699669 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.699629 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f667fce-363f-476e-8601-afd78a7f3121-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f667fce-363f-476e-8601-afd78a7f3121-istio-podinfo\") pod \"0f667fce-363f-476e-8601-afd78a7f3121\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " Nov 03 20:22:07.699669 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.699672 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f667fce-363f-476e-8601-afd78a7f3121-istio-token\" (UniqueName: \"kubernetes.io/projected/0f667fce-363f-476e-8601-afd78a7f3121-istio-token\") pod \"0f667fce-363f-476e-8601-afd78a7f3121\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " Nov 03 20:22:07.699970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.699794 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-workload-socket\") pod \"0f667fce-363f-476e-8601-afd78a7f3121\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " Nov 03 20:22:07.699970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.699881 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-istio-data\") pod \"0f667fce-363f-476e-8601-afd78a7f3121\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " Nov 03 20:22:07.699970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.699925 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f667fce-363f-476e-8601-afd78a7f3121-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f667fce-363f-476e-8601-afd78a7f3121-istiod-ca-cert\") pod \"0f667fce-363f-476e-8601-afd78a7f3121\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " Nov 03 20:22:07.700147 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.700016 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-workload-certs\") pod \"0f667fce-363f-476e-8601-afd78a7f3121\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " Nov 03 20:22:07.700147 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.700058 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f667fce-363f-476e-8601-afd78a7f3121-kube-api-access-895wn\" (UniqueName: \"kubernetes.io/projected/0f667fce-363f-476e-8601-afd78a7f3121-kube-api-access-895wn\") pod \"0f667fce-363f-476e-8601-afd78a7f3121\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " Nov 03 20:22:07.700262 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.700146 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-istio-envoy\") pod \"0f667fce-363f-476e-8601-afd78a7f3121\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " Nov 03 20:22:07.700262 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.700199 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-credential-socket\") pod \"0f667fce-363f-476e-8601-afd78a7f3121\" (UID: \"0f667fce-363f-476e-8601-afd78a7f3121\") " Nov 03 20:22:07.701073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.700752 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-workload-socket" pod "0f667fce-363f-476e-8601-afd78a7f3121" (UID: "0f667fce-363f-476e-8601-afd78a7f3121"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:07.701073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.700789 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-istio-data" pod "0f667fce-363f-476e-8601-afd78a7f3121" (UID: "0f667fce-363f-476e-8601-afd78a7f3121"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:07.701073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.700980 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-credential-socket" pod "0f667fce-363f-476e-8601-afd78a7f3121" (UID: "0f667fce-363f-476e-8601-afd78a7f3121"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:07.701073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.701044 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:07.701073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.701067 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:07.701489 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.701465 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-workload-certs" pod "0f667fce-363f-476e-8601-afd78a7f3121" (UID: "0f667fce-363f-476e-8601-afd78a7f3121"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:07.701690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.701660 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f667fce-363f-476e-8601-afd78a7f3121-istiod-ca-cert" pod "0f667fce-363f-476e-8601-afd78a7f3121" (UID: "0f667fce-363f-476e-8601-afd78a7f3121"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:07.704214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.704179 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f667fce-363f-476e-8601-afd78a7f3121-istio-podinfo" pod "0f667fce-363f-476e-8601-afd78a7f3121" (UID: "0f667fce-363f-476e-8601-afd78a7f3121"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:22:07.710985 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.710389 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f667fce-363f-476e-8601-afd78a7f3121-kube-api-access-895wn" pod "0f667fce-363f-476e-8601-afd78a7f3121" (UID: "0f667fce-363f-476e-8601-afd78a7f3121"). InnerVolumeSpecName "kube-api-access-895wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:07.711101 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.711059 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-istio-envoy" pod "0f667fce-363f-476e-8601-afd78a7f3121" (UID: "0f667fce-363f-476e-8601-afd78a7f3121"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:07.711312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.711283 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f667fce-363f-476e-8601-afd78a7f3121-istio-token" pod "0f667fce-363f-476e-8601-afd78a7f3121" (UID: "0f667fce-363f-476e-8601-afd78a7f3121"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:07.802319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.802283 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:07.802319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.802315 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-895wn\" (UniqueName: \"kubernetes.io/projected/0f667fce-363f-476e-8601-afd78a7f3121-kube-api-access-895wn\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:07.802319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.802325 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:07.802848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.802334 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f667fce-363f-476e-8601-afd78a7f3121-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:07.802848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.802344 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f667fce-363f-476e-8601-afd78a7f3121-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:07.802848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.802352 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f667fce-363f-476e-8601-afd78a7f3121-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:07.802848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:07.802362 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f667fce-363f-476e-8601-afd78a7f3121-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:08.228752 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:08.228714 5111 generic.go:356] "Generic (PLEG): container finished" podID="0f667fce-363f-476e-8601-afd78a7f3121" containerID="fe9bef312215087c89b4e4f4e014ecc2813ca74b37f58c5b67a1d092cacf5aa4" exitCode=0 Nov 03 20:22:08.228934 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:08.228770 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" Nov 03 20:22:08.228934 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:08.228792 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" event={"ID":"0f667fce-363f-476e-8601-afd78a7f3121","Type":"ContainerDied","Data":"fe9bef312215087c89b4e4f4e014ecc2813ca74b37f58c5b67a1d092cacf5aa4"} Nov 03 20:22:08.228934 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:08.228856 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c" event={"ID":"0f667fce-363f-476e-8601-afd78a7f3121","Type":"ContainerDied","Data":"d856bdbe3c6ef4c3164beb346b1a0a4b4c1f2d668e07b3df92609688cfef4b07"} Nov 03 20:22:08.228934 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:08.228886 5111 scope.go:122] "RemoveContainer" containerID="fe9bef312215087c89b4e4f4e014ecc2813ca74b37f58c5b67a1d092cacf5aa4" Nov 03 20:22:08.244605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:08.244579 5111 scope.go:122] "RemoveContainer" containerID="556de97197debf90d7ad5b1f2188cfa171a2e726d25c11fb54e706de9ecdb3b0" Nov 03 20:22:08.249805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:08.249780 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c"] Nov 03 20:22:08.254643 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:08.254621 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-p994c"] Nov 03 20:22:08.265366 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:08.265344 5111 scope.go:122] "RemoveContainer" containerID="fe9bef312215087c89b4e4f4e014ecc2813ca74b37f58c5b67a1d092cacf5aa4" Nov 03 20:22:08.265770 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:22:08.265746 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9bef312215087c89b4e4f4e014ecc2813ca74b37f58c5b67a1d092cacf5aa4\": container with ID starting with fe9bef312215087c89b4e4f4e014ecc2813ca74b37f58c5b67a1d092cacf5aa4 not found: ID does not exist" containerID="fe9bef312215087c89b4e4f4e014ecc2813ca74b37f58c5b67a1d092cacf5aa4" Nov 03 20:22:08.265863 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:08.265777 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9bef312215087c89b4e4f4e014ecc2813ca74b37f58c5b67a1d092cacf5aa4"} err="failed to get container status \"fe9bef312215087c89b4e4f4e014ecc2813ca74b37f58c5b67a1d092cacf5aa4\": rpc error: code = NotFound desc = could not find container \"fe9bef312215087c89b4e4f4e014ecc2813ca74b37f58c5b67a1d092cacf5aa4\": container with ID starting with fe9bef312215087c89b4e4f4e014ecc2813ca74b37f58c5b67a1d092cacf5aa4 not found: ID does not exist" Nov 03 20:22:08.265863 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:08.265800 5111 scope.go:122] "RemoveContainer" containerID="556de97197debf90d7ad5b1f2188cfa171a2e726d25c11fb54e706de9ecdb3b0" Nov 03 20:22:08.266121 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:22:08.266096 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556de97197debf90d7ad5b1f2188cfa171a2e726d25c11fb54e706de9ecdb3b0\": container with ID starting with 556de97197debf90d7ad5b1f2188cfa171a2e726d25c11fb54e706de9ecdb3b0 not found: ID does not exist" containerID="556de97197debf90d7ad5b1f2188cfa171a2e726d25c11fb54e706de9ecdb3b0" Nov 03 20:22:08.266194 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:08.266131 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556de97197debf90d7ad5b1f2188cfa171a2e726d25c11fb54e706de9ecdb3b0"} err="failed to get container status \"556de97197debf90d7ad5b1f2188cfa171a2e726d25c11fb54e706de9ecdb3b0\": rpc error: code = NotFound desc = could not find container \"556de97197debf90d7ad5b1f2188cfa171a2e726d25c11fb54e706de9ecdb3b0\": container with ID starting with 556de97197debf90d7ad5b1f2188cfa171a2e726d25c11fb54e706de9ecdb3b0 not found: ID does not exist" Nov 03 20:22:10.201671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:10.201633 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f667fce-363f-476e-8601-afd78a7f3121" path="/var/lib/kubelet/pods/0f667fce-363f-476e-8601-afd78a7f3121/volumes" Nov 03 20:22:24.259700 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:24.259648 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:22:24.260199 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:24.259713 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:25.475982 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:25.475948 5111 generic.go:356] "Generic (PLEG): container finished" podID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerID="f237b247cefe086452423efa17b16ea60f036937087d5c6d50da4f16ebb0448b" exitCode=0 Nov 03 20:22:25.476783 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:25.476025 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" event={"ID":"2815d45f-e02c-4178-9f95-0375fda5f11c","Type":"ContainerDied","Data":"f237b247cefe086452423efa17b16ea60f036937087d5c6d50da4f16ebb0448b"} Nov 03 20:22:25.476783 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:25.476636 5111 scope.go:122] "RemoveContainer" containerID="f237b247cefe086452423efa17b16ea60f036937087d5c6d50da4f16ebb0448b" Nov 03 20:22:25.478675 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:25.478625 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:22:25.478815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:25.478737 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:22:25.478815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:25.478786 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:22:26.486457 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:26.486421 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" event={"ID":"2815d45f-e02c-4178-9f95-0375fda5f11c","Type":"ContainerStarted","Data":"545d1e185fbc992f113b4d2ef69728ab28f974921050e940df2c0971123affca"} Nov 03 20:22:27.274556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:27.274463 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:22:27.274780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:27.274582 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 20:22:27.274780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:27.274625 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 20:22:28.275008 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:28.274970 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 20:22:28.275439 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:28.275028 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 20:22:29.274076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:29.274031 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 20:22:29.274282 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:29.274098 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 20:22:30.276584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:30.274966 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 20:22:30.276584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:30.275032 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 20:22:31.274572 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:31.274491 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 20:22:31.274789 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:31.274591 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 20:22:32.274140 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:32.274096 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 20:22:32.274845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:32.274803 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 20:22:33.278880 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:33.278850 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:22:33.279376 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:33.279357 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:22:33.280136 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:33.280117 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:22:35.062100 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:22:35.062064 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:22:47.542705 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.542669 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8"] Nov 03 20:22:47.543515 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.543499 5111 cpu_manager.go:414] "containerMap: removing container" podUID="0f667fce-363f-476e-8601-afd78a7f3121" containerName="istio-proxy" Nov 03 20:22:47.543587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.543518 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f667fce-363f-476e-8601-afd78a7f3121" containerName="istio-proxy" Nov 03 20:22:47.543587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.543552 5111 cpu_manager.go:414] "containerMap: removing container" podUID="0f667fce-363f-476e-8601-afd78a7f3121" containerName="istio-proxy" Nov 03 20:22:47.543587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.543558 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f667fce-363f-476e-8601-afd78a7f3121" containerName="istio-proxy" Nov 03 20:22:47.543729 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.543718 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f667fce-363f-476e-8601-afd78a7f3121" containerName="istio-proxy" Nov 03 20:22:47.543769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.543730 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f667fce-363f-476e-8601-afd78a7f3121" containerName="istio-proxy" Nov 03 20:22:47.549901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.549875 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" Nov 03 20:22:47.554994 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.554962 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8"] Nov 03 20:22:47.558243 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.558199 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:22:47.558403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.558276 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:22:47.731281 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.731236 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nz94\" (UniqueName: \"kubernetes.io/projected/cbf2b76c-c679-4c21-964c-a6c0361102f2-kube-api-access-9nz94\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8\" (UID: \"cbf2b76c-c679-4c21-964c-a6c0361102f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" Nov 03 20:22:47.731997 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.731961 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf2b76c-c679-4c21-964c-a6c0361102f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8\" (UID: \"cbf2b76c-c679-4c21-964c-a6c0361102f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" Nov 03 20:22:47.732236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.732215 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf2b76c-c679-4c21-964c-a6c0361102f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8\" (UID: \"cbf2b76c-c679-4c21-964c-a6c0361102f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" Nov 03 20:22:47.838276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.838182 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf2b76c-c679-4c21-964c-a6c0361102f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8\" (UID: \"cbf2b76c-c679-4c21-964c-a6c0361102f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" Nov 03 20:22:47.838276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.838248 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf2b76c-c679-4c21-964c-a6c0361102f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8\" (UID: \"cbf2b76c-c679-4c21-964c-a6c0361102f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" Nov 03 20:22:47.838484 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.838372 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nz94\" (UniqueName: \"kubernetes.io/projected/cbf2b76c-c679-4c21-964c-a6c0361102f2-kube-api-access-9nz94\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8\" (UID: \"cbf2b76c-c679-4c21-964c-a6c0361102f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" Nov 03 20:22:47.839034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.838998 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf2b76c-c679-4c21-964c-a6c0361102f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8\" (UID: \"cbf2b76c-c679-4c21-964c-a6c0361102f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" Nov 03 20:22:47.840746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.840722 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf2b76c-c679-4c21-964c-a6c0361102f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8\" (UID: \"cbf2b76c-c679-4c21-964c-a6c0361102f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" Nov 03 20:22:47.845987 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.845965 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nz94\" (UniqueName: \"kubernetes.io/projected/cbf2b76c-c679-4c21-964c-a6c0361102f2-kube-api-access-9nz94\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8\" (UID: \"cbf2b76c-c679-4c21-964c-a6c0361102f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" Nov 03 20:22:47.875867 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:47.875826 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" Nov 03 20:22:48.162448 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:48.162417 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8"] Nov 03 20:22:48.162981 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:22:48.162949 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf2b76c_c679_4c21_964c_a6c0361102f2.slice/crio-4998a8eec8cc29d25351f91e244e0d24ffdf87d13e13b8c30840d9799981fd6d WatchSource:0}: Error finding container 4998a8eec8cc29d25351f91e244e0d24ffdf87d13e13b8c30840d9799981fd6d: Status 404 returned error can't find the container with id 4998a8eec8cc29d25351f91e244e0d24ffdf87d13e13b8c30840d9799981fd6d Nov 03 20:22:48.867245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:48.867158 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" event={"ID":"cbf2b76c-c679-4c21-964c-a6c0361102f2","Type":"ContainerStarted","Data":"242f4e9e3f8f8e34edb84ac368f94923bb8a5e0841b69043bc882911042f34db"} Nov 03 20:22:48.867245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:48.867236 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" event={"ID":"cbf2b76c-c679-4c21-964c-a6c0361102f2","Type":"ContainerStarted","Data":"4998a8eec8cc29d25351f91e244e0d24ffdf87d13e13b8c30840d9799981fd6d"} Nov 03 20:22:48.886000 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:48.885948 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" podStartSLOduration=1.885934751 podStartE2EDuration="1.885934751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:48.881575681 +0000 UTC m=+2598.843954119" watchObservedRunningTime="2026-11-03 20:22:48.885934751 +0000 UTC m=+2598.848313189" Nov 03 20:22:49.877864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:49.877826 5111 generic.go:356] "Generic (PLEG): container finished" podID="cbf2b76c-c679-4c21-964c-a6c0361102f2" containerID="242f4e9e3f8f8e34edb84ac368f94923bb8a5e0841b69043bc882911042f34db" exitCode=0 Nov 03 20:22:49.878282 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:49.877915 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" event={"ID":"cbf2b76c-c679-4c21-964c-a6c0361102f2","Type":"ContainerDied","Data":"242f4e9e3f8f8e34edb84ac368f94923bb8a5e0841b69043bc882911042f34db"} Nov 03 20:22:51.141222 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:51.141148 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" Nov 03 20:22:51.236925 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:51.236402 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf2b76c-c679-4c21-964c-a6c0361102f2-kube-api-access-9nz94\" (UniqueName: \"kubernetes.io/projected/cbf2b76c-c679-4c21-964c-a6c0361102f2-kube-api-access-9nz94\") pod \"cbf2b76c-c679-4c21-964c-a6c0361102f2\" (UID: \"cbf2b76c-c679-4c21-964c-a6c0361102f2\") " Nov 03 20:22:51.236925 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:51.236906 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf2b76c-c679-4c21-964c-a6c0361102f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf2b76c-c679-4c21-964c-a6c0361102f2-maas-api-openshift-ca\") pod \"cbf2b76c-c679-4c21-964c-a6c0361102f2\" (UID: \"cbf2b76c-c679-4c21-964c-a6c0361102f2\") " Nov 03 20:22:51.237180 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:51.237074 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbf2b76c-c679-4c21-964c-a6c0361102f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf2b76c-c679-4c21-964c-a6c0361102f2-maas-api-xks-ca\") pod \"cbf2b76c-c679-4c21-964c-a6c0361102f2\" (UID: \"cbf2b76c-c679-4c21-964c-a6c0361102f2\") " Nov 03 20:22:51.237474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:51.237443 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf2b76c-c679-4c21-964c-a6c0361102f2-maas-api-openshift-ca" pod "cbf2b76c-c679-4c21-964c-a6c0361102f2" (UID: "cbf2b76c-c679-4c21-964c-a6c0361102f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:51.238197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:51.238173 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbf2b76c-c679-4c21-964c-a6c0361102f2-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:51.239192 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:51.239167 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf2b76c-c679-4c21-964c-a6c0361102f2-kube-api-access-9nz94" pod "cbf2b76c-c679-4c21-964c-a6c0361102f2" (UID: "cbf2b76c-c679-4c21-964c-a6c0361102f2"). InnerVolumeSpecName "kube-api-access-9nz94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:51.239795 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:51.239764 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf2b76c-c679-4c21-964c-a6c0361102f2-maas-api-xks-ca" pod "cbf2b76c-c679-4c21-964c-a6c0361102f2" (UID: "cbf2b76c-c679-4c21-964c-a6c0361102f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:51.340367 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:51.340328 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nz94\" (UniqueName: \"kubernetes.io/projected/cbf2b76c-c679-4c21-964c-a6c0361102f2-kube-api-access-9nz94\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:51.340367 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:51.340363 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbf2b76c-c679-4c21-964c-a6c0361102f2-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:51.897926 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:51.897893 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" Nov 03 20:22:51.898114 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:51.897900 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8" event={"ID":"cbf2b76c-c679-4c21-964c-a6c0361102f2","Type":"ContainerDied","Data":"4998a8eec8cc29d25351f91e244e0d24ffdf87d13e13b8c30840d9799981fd6d"} Nov 03 20:22:51.898114 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:22:51.898048 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4998a8eec8cc29d25351f91e244e0d24ffdf87d13e13b8c30840d9799981fd6d" Nov 03 20:23:04.940959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:04.940910 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8"] Nov 03 20:23:04.945442 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:04.945412 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9ff9f5-696z8"] Nov 03 20:23:05.323914 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:23:05.323808 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:23:06.188034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:06.188000 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf2b76c-c679-4c21-964c-a6c0361102f2" path="/var/lib/kubelet/pods/cbf2b76c-c679-4c21-964c-a6c0361102f2/volumes" Nov 03 20:23:12.086037 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.085998 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc"] Nov 03 20:23:12.087069 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.087048 5111 cpu_manager.go:414] "containerMap: removing container" podUID="cbf2b76c-c679-4c21-964c-a6c0361102f2" containerName="revoke-keys" Nov 03 20:23:12.087069 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.087069 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf2b76c-c679-4c21-964c-a6c0361102f2" containerName="revoke-keys" Nov 03 20:23:12.087224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.087215 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf2b76c-c679-4c21-964c-a6c0361102f2" containerName="revoke-keys" Nov 03 20:23:12.092116 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.091707 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" Nov 03 20:23:12.096303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.096276 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:23:12.099739 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.099712 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc"] Nov 03 20:23:12.099930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.099911 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:23:12.202400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.202366 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687bac8e-f566-4848-8dd7-2572ae2018b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc\" (UID: \"687bac8e-f566-4848-8dd7-2572ae2018b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" Nov 03 20:23:12.202683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.202617 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9lwm\" (UniqueName: \"kubernetes.io/projected/687bac8e-f566-4848-8dd7-2572ae2018b0-kube-api-access-c9lwm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc\" (UID: \"687bac8e-f566-4848-8dd7-2572ae2018b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" Nov 03 20:23:12.202859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.202843 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687bac8e-f566-4848-8dd7-2572ae2018b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc\" (UID: \"687bac8e-f566-4848-8dd7-2572ae2018b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" Nov 03 20:23:12.304183 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.304147 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687bac8e-f566-4848-8dd7-2572ae2018b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc\" (UID: \"687bac8e-f566-4848-8dd7-2572ae2018b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" Nov 03 20:23:12.304377 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.304228 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687bac8e-f566-4848-8dd7-2572ae2018b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc\" (UID: \"687bac8e-f566-4848-8dd7-2572ae2018b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" Nov 03 20:23:12.304377 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.304373 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9lwm\" (UniqueName: \"kubernetes.io/projected/687bac8e-f566-4848-8dd7-2572ae2018b0-kube-api-access-c9lwm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc\" (UID: \"687bac8e-f566-4848-8dd7-2572ae2018b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" Nov 03 20:23:12.304917 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.304893 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687bac8e-f566-4848-8dd7-2572ae2018b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc\" (UID: \"687bac8e-f566-4848-8dd7-2572ae2018b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" Nov 03 20:23:12.306557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.306526 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687bac8e-f566-4848-8dd7-2572ae2018b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc\" (UID: \"687bac8e-f566-4848-8dd7-2572ae2018b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" Nov 03 20:23:12.312391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.312363 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9lwm\" (UniqueName: \"kubernetes.io/projected/687bac8e-f566-4848-8dd7-2572ae2018b0-kube-api-access-c9lwm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc\" (UID: \"687bac8e-f566-4848-8dd7-2572ae2018b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" Nov 03 20:23:12.415980 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.415947 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" Nov 03 20:23:12.708965 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:12.708937 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc"] Nov 03 20:23:12.709083 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:23:12.708992 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687bac8e_f566_4848_8dd7_2572ae2018b0.slice/crio-c671ac3a7bec987b091d3ed85bced9820978d5f4972640bba3190dd564b04eab WatchSource:0}: Error finding container c671ac3a7bec987b091d3ed85bced9820978d5f4972640bba3190dd564b04eab: Status 404 returned error can't find the container with id c671ac3a7bec987b091d3ed85bced9820978d5f4972640bba3190dd564b04eab Nov 03 20:23:13.330258 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:13.330167 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" event={"ID":"687bac8e-f566-4848-8dd7-2572ae2018b0","Type":"ContainerStarted","Data":"eb75b9b471554f04ab33a3465e345ebc4f9e9c982989167066b30cb11dab67d2"} Nov 03 20:23:13.330875 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:13.330848 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" event={"ID":"687bac8e-f566-4848-8dd7-2572ae2018b0","Type":"ContainerStarted","Data":"c671ac3a7bec987b091d3ed85bced9820978d5f4972640bba3190dd564b04eab"} Nov 03 20:23:13.354289 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:13.354231 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" podStartSLOduration=1.354215381 podStartE2EDuration="1.354215381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:13.349653614 +0000 UTC m=+2623.312032069" watchObservedRunningTime="2026-11-03 20:23:13.354215381 +0000 UTC m=+2623.316593819" Nov 03 20:23:14.340456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:14.340422 5111 generic.go:356] "Generic (PLEG): container finished" podID="687bac8e-f566-4848-8dd7-2572ae2018b0" containerID="eb75b9b471554f04ab33a3465e345ebc4f9e9c982989167066b30cb11dab67d2" exitCode=0 Nov 03 20:23:14.340904 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:14.340504 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" event={"ID":"687bac8e-f566-4848-8dd7-2572ae2018b0","Type":"ContainerDied","Data":"eb75b9b471554f04ab33a3465e345ebc4f9e9c982989167066b30cb11dab67d2"} Nov 03 20:23:15.560286 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:15.560256 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" Nov 03 20:23:15.676654 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:15.676614 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/687bac8e-f566-4848-8dd7-2572ae2018b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687bac8e-f566-4848-8dd7-2572ae2018b0-maas-api-openshift-ca\") pod \"687bac8e-f566-4848-8dd7-2572ae2018b0\" (UID: \"687bac8e-f566-4848-8dd7-2572ae2018b0\") " Nov 03 20:23:15.676858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:15.676719 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/687bac8e-f566-4848-8dd7-2572ae2018b0-kube-api-access-c9lwm\" (UniqueName: \"kubernetes.io/projected/687bac8e-f566-4848-8dd7-2572ae2018b0-kube-api-access-c9lwm\") pod \"687bac8e-f566-4848-8dd7-2572ae2018b0\" (UID: \"687bac8e-f566-4848-8dd7-2572ae2018b0\") " Nov 03 20:23:15.676858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:15.676830 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/687bac8e-f566-4848-8dd7-2572ae2018b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687bac8e-f566-4848-8dd7-2572ae2018b0-maas-api-xks-ca\") pod \"687bac8e-f566-4848-8dd7-2572ae2018b0\" (UID: \"687bac8e-f566-4848-8dd7-2572ae2018b0\") " Nov 03 20:23:15.677084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:15.677051 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687bac8e-f566-4848-8dd7-2572ae2018b0-maas-api-openshift-ca" pod "687bac8e-f566-4848-8dd7-2572ae2018b0" (UID: "687bac8e-f566-4848-8dd7-2572ae2018b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:15.677516 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:15.677498 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687bac8e-f566-4848-8dd7-2572ae2018b0-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:15.679008 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:15.678980 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687bac8e-f566-4848-8dd7-2572ae2018b0-kube-api-access-c9lwm" pod "687bac8e-f566-4848-8dd7-2572ae2018b0" (UID: "687bac8e-f566-4848-8dd7-2572ae2018b0"). InnerVolumeSpecName "kube-api-access-c9lwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:15.679126 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:15.679070 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/687bac8e-f566-4848-8dd7-2572ae2018b0-maas-api-xks-ca" pod "687bac8e-f566-4848-8dd7-2572ae2018b0" (UID: "687bac8e-f566-4848-8dd7-2572ae2018b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:15.779363 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:15.779319 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9lwm\" (UniqueName: \"kubernetes.io/projected/687bac8e-f566-4848-8dd7-2572ae2018b0-kube-api-access-c9lwm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:15.779363 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:15.779360 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687bac8e-f566-4848-8dd7-2572ae2018b0-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:16.366622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:16.366586 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" Nov 03 20:23:16.366622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:16.366596 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc" event={"ID":"687bac8e-f566-4848-8dd7-2572ae2018b0","Type":"ContainerDied","Data":"c671ac3a7bec987b091d3ed85bced9820978d5f4972640bba3190dd564b04eab"} Nov 03 20:23:16.366893 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:16.366672 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c671ac3a7bec987b091d3ed85bced9820978d5f4972640bba3190dd564b04eab" Nov 03 20:23:19.600718 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:19.600670 5111 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r" podUID="e52a6c08-4959-4cf8-90a6-fe48e0dbd12b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded" Nov 03 20:23:22.258814 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:22.252323 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8"] Nov 03 20:23:27.253115 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.253070 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" podUID="9792b23b-9450-44e3-bb6c-c90a9d5411fa" containerName="payload-processing" containerID="cri-o://3064cf0c4d5b2cb92d0c5dd509ea4ca959e99e1d97f06afb22784614348b697b" gracePeriod=25 Nov 03 20:23:27.592267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.592240 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" Nov 03 20:23:27.647989 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.647943 5111 generic.go:356] "Generic (PLEG): container finished" podID="9792b23b-9450-44e3-bb6c-c90a9d5411fa" containerID="3064cf0c4d5b2cb92d0c5dd509ea4ca959e99e1d97f06afb22784614348b697b" exitCode=0 Nov 03 20:23:27.648193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.647996 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" Nov 03 20:23:27.648193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.648032 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" event={"ID":"9792b23b-9450-44e3-bb6c-c90a9d5411fa","Type":"ContainerDied","Data":"3064cf0c4d5b2cb92d0c5dd509ea4ca959e99e1d97f06afb22784614348b697b"} Nov 03 20:23:27.648193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.648102 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8" event={"ID":"9792b23b-9450-44e3-bb6c-c90a9d5411fa","Type":"ContainerDied","Data":"9a278df1da708879a0cde4068fe0a32532be57e3c7b90e441f8ea7fceec338e3"} Nov 03 20:23:27.648193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.648131 5111 scope.go:122] "RemoveContainer" containerID="3064cf0c4d5b2cb92d0c5dd509ea4ca959e99e1d97f06afb22784614348b697b" Nov 03 20:23:27.662088 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.662045 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9792b23b-9450-44e3-bb6c-c90a9d5411fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9792b23b-9450-44e3-bb6c-c90a9d5411fa-plugins-config-volume\") pod \"9792b23b-9450-44e3-bb6c-c90a9d5411fa\" (UID: \"9792b23b-9450-44e3-bb6c-c90a9d5411fa\") " Nov 03 20:23:27.662165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.662107 5111 scope.go:122] "RemoveContainer" containerID="3064cf0c4d5b2cb92d0c5dd509ea4ca959e99e1d97f06afb22784614348b697b" Nov 03 20:23:27.662293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.662269 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9792b23b-9450-44e3-bb6c-c90a9d5411fa-kube-api-access-7m9mc\" (UniqueName: \"kubernetes.io/projected/9792b23b-9450-44e3-bb6c-c90a9d5411fa-kube-api-access-7m9mc\") pod \"9792b23b-9450-44e3-bb6c-c90a9d5411fa\" (UID: \"9792b23b-9450-44e3-bb6c-c90a9d5411fa\") " Nov 03 20:23:27.662477 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:23:27.662452 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3064cf0c4d5b2cb92d0c5dd509ea4ca959e99e1d97f06afb22784614348b697b\": container with ID starting with 3064cf0c4d5b2cb92d0c5dd509ea4ca959e99e1d97f06afb22784614348b697b not found: ID does not exist" containerID="3064cf0c4d5b2cb92d0c5dd509ea4ca959e99e1d97f06afb22784614348b697b" Nov 03 20:23:27.662609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.662488 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3064cf0c4d5b2cb92d0c5dd509ea4ca959e99e1d97f06afb22784614348b697b"} err="failed to get container status \"3064cf0c4d5b2cb92d0c5dd509ea4ca959e99e1d97f06afb22784614348b697b\": rpc error: code = NotFound desc = could not find container \"3064cf0c4d5b2cb92d0c5dd509ea4ca959e99e1d97f06afb22784614348b697b\": container with ID starting with 3064cf0c4d5b2cb92d0c5dd509ea4ca959e99e1d97f06afb22784614348b697b not found: ID does not exist" Nov 03 20:23:27.662859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.662830 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9792b23b-9450-44e3-bb6c-c90a9d5411fa-plugins-config-volume" pod "9792b23b-9450-44e3-bb6c-c90a9d5411fa" (UID: "9792b23b-9450-44e3-bb6c-c90a9d5411fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:27.663255 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.663233 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9792b23b-9450-44e3-bb6c-c90a9d5411fa-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:27.664566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.664543 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9792b23b-9450-44e3-bb6c-c90a9d5411fa-kube-api-access-7m9mc" pod "9792b23b-9450-44e3-bb6c-c90a9d5411fa" (UID: "9792b23b-9450-44e3-bb6c-c90a9d5411fa"). InnerVolumeSpecName "kube-api-access-7m9mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:27.764782 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.764745 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m9mc\" (UniqueName: \"kubernetes.io/projected/9792b23b-9450-44e3-bb6c-c90a9d5411fa-kube-api-access-7m9mc\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:27.979647 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.979610 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8"] Nov 03 20:23:27.984018 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:27.983991 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a8e19e-7b76756bfd-npjt8"] Nov 03 20:23:28.185171 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:28.185142 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9792b23b-9450-44e3-bb6c-c90a9d5411fa" path="/var/lib/kubelet/pods/9792b23b-9450-44e3-bb6c-c90a9d5411fa/volumes" Nov 03 20:23:31.614170 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:31.612589 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc"] Nov 03 20:23:31.624510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:31.624464 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e19e-kl2gc"] Nov 03 20:23:32.187367 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:32.187329 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="687bac8e-f566-4848-8dd7-2572ae2018b0" path="/var/lib/kubelet/pods/687bac8e-f566-4848-8dd7-2572ae2018b0/volumes" Nov 03 20:23:34.576844 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.576800 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr"] Nov 03 20:23:34.577840 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.577810 5111 cpu_manager.go:414] "containerMap: removing container" podUID="687bac8e-f566-4848-8dd7-2572ae2018b0" containerName="revoke-keys" Nov 03 20:23:34.577840 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.577839 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="687bac8e-f566-4848-8dd7-2572ae2018b0" containerName="revoke-keys" Nov 03 20:23:34.578022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.577904 5111 cpu_manager.go:414] "containerMap: removing container" podUID="9792b23b-9450-44e3-bb6c-c90a9d5411fa" containerName="payload-processing" Nov 03 20:23:34.578022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.577912 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9792b23b-9450-44e3-bb6c-c90a9d5411fa" containerName="payload-processing" Nov 03 20:23:34.578164 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.578100 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="9792b23b-9450-44e3-bb6c-c90a9d5411fa" containerName="payload-processing" Nov 03 20:23:34.578164 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.578126 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="687bac8e-f566-4848-8dd7-2572ae2018b0" containerName="revoke-keys" Nov 03 20:23:34.582302 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.582277 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.595713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.595684 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr"] Nov 03 20:23:34.725629 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.725591 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.725629 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.725632 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28da3b98-e5f9-4355-876f-560f633061de-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.725874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.725741 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.725874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.725771 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.725874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.725855 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28da3b98-e5f9-4355-876f-560f633061de-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.725988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.725897 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.725988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.725943 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.726149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.726121 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fddjp\" (UniqueName: \"kubernetes.io/projected/28da3b98-e5f9-4355-876f-560f633061de-kube-api-access-fddjp\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.726224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.726179 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28da3b98-e5f9-4355-876f-560f633061de-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.830333 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.830237 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.830333 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.830293 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28da3b98-e5f9-4355-876f-560f633061de-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.830724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.830353 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.830724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.830404 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.830724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.830455 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28da3b98-e5f9-4355-876f-560f633061de-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.830724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.830494 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.831062 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.831039 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.831216 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.831196 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.832427 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.832405 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fddjp\" (UniqueName: \"kubernetes.io/projected/28da3b98-e5f9-4355-876f-560f633061de-kube-api-access-fddjp\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.832620 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.832599 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28da3b98-e5f9-4355-876f-560f633061de-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.832897 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.832837 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28da3b98-e5f9-4355-876f-560f633061de-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.832897 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.831737 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.832897 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.831806 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.833156 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.832055 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.846946 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.846910 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.847984 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.847957 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28da3b98-e5f9-4355-876f-560f633061de-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.850233 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.850205 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fddjp\" (UniqueName: \"kubernetes.io/projected/28da3b98-e5f9-4355-876f-560f633061de-kube-api-access-fddjp\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.850391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.850291 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28da3b98-e5f9-4355-876f-560f633061de-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:34.901543 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:34.901507 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:35.074766 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:23:35.074728 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:23:35.191481 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:35.191446 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr"] Nov 03 20:23:35.191970 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:23:35.191938 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28da3b98_e5f9_4355_876f_560f633061de.slice/crio-a341df54eb7e9761e398160a15c3edb60595b88aba515a7f612ff12d2075e601 WatchSource:0}: Error finding container a341df54eb7e9761e398160a15c3edb60595b88aba515a7f612ff12d2075e601: Status 404 returned error can't find the container with id a341df54eb7e9761e398160a15c3edb60595b88aba515a7f612ff12d2075e601 Nov 03 20:23:35.194829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:35.194793 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:23:35.194921 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:35.194861 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:23:35.194921 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:35.194893 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:23:35.743486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:35.743446 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" event={"ID":"28da3b98-e5f9-4355-876f-560f633061de","Type":"ContainerStarted","Data":"308dfd08165678b1b04a95906e90078ecdbf6c87a114bfc8e4ebd0c4b608d59a"} Nov 03 20:23:35.743967 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:35.743512 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" event={"ID":"28da3b98-e5f9-4355-876f-560f633061de","Type":"ContainerStarted","Data":"a341df54eb7e9761e398160a15c3edb60595b88aba515a7f612ff12d2075e601"} Nov 03 20:23:35.763031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:35.762971 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" podStartSLOduration=1.762955292 podStartE2EDuration="1.762955292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:35.76105897 +0000 UTC m=+2645.723437435" watchObservedRunningTime="2026-11-03 20:23:35.762955292 +0000 UTC m=+2645.725333732" Nov 03 20:23:35.902372 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:35.902326 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:36.903217 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:36.903183 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 03 20:23:36.903864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:36.903242 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" podUID="28da3b98-e5f9-4355-876f-560f633061de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 03 20:23:37.901894 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:37.901850 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 03 20:23:37.902081 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:37.901908 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" podUID="28da3b98-e5f9-4355-876f-560f633061de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 03 20:23:38.826170 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:38.826137 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq"] Nov 03 20:23:38.833123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:38.833098 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq" Nov 03 20:23:38.837129 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:38.837103 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:23:38.837576 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:38.837549 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq"] Nov 03 20:23:38.837901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:38.837880 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:23:38.903432 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:38.903392 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 03 20:23:38.903690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:38.903456 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" podUID="28da3b98-e5f9-4355-876f-560f633061de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 03 20:23:38.927068 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:38.927017 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff994a2-65e2-44da-8274-46384f9b1971-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq\" (UID: \"4ff994a2-65e2-44da-8274-46384f9b1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq" Nov 03 20:23:38.927261 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:38.927109 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff994a2-65e2-44da-8274-46384f9b1971-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq\" (UID: \"4ff994a2-65e2-44da-8274-46384f9b1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq" Nov 03 20:23:38.927261 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:38.927177 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmrz6\" (UniqueName: \"kubernetes.io/projected/4ff994a2-65e2-44da-8274-46384f9b1971-kube-api-access-dmrz6\") pod \"maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq\" (UID: \"4ff994a2-65e2-44da-8274-46384f9b1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq" Nov 03 20:23:39.029759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:39.029721 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff994a2-65e2-44da-8274-46384f9b1971-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq\" (UID: \"4ff994a2-65e2-44da-8274-46384f9b1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq" Nov 03 20:23:39.029975 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:39.029795 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff994a2-65e2-44da-8274-46384f9b1971-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq\" (UID: \"4ff994a2-65e2-44da-8274-46384f9b1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq" Nov 03 20:23:39.029975 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:39.029860 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmrz6\" (UniqueName: \"kubernetes.io/projected/4ff994a2-65e2-44da-8274-46384f9b1971-kube-api-access-dmrz6\") pod \"maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq\" (UID: \"4ff994a2-65e2-44da-8274-46384f9b1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq" Nov 03 20:23:39.030599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:39.030519 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff994a2-65e2-44da-8274-46384f9b1971-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq\" (UID: \"4ff994a2-65e2-44da-8274-46384f9b1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq" Nov 03 20:23:39.032465 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:39.032442 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff994a2-65e2-44da-8274-46384f9b1971-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq\" (UID: \"4ff994a2-65e2-44da-8274-46384f9b1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq" Nov 03 20:23:39.044214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:39.044172 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmrz6\" (UniqueName: \"kubernetes.io/projected/4ff994a2-65e2-44da-8274-46384f9b1971-kube-api-access-dmrz6\") pod \"maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq\" (UID: \"4ff994a2-65e2-44da-8274-46384f9b1971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq" Nov 03 20:23:39.189516 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:39.189488 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq" Nov 03 20:23:39.504068 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:39.504040 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq"] Nov 03 20:23:39.504292 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:23:39.504261 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff994a2_65e2_44da_8274_46384f9b1971.slice/crio-fe163e0b70e3d7259ed9945f97305662bc4fab3f4370d9942e0a1fe7e831d834 WatchSource:0}: Error finding container fe163e0b70e3d7259ed9945f97305662bc4fab3f4370d9942e0a1fe7e831d834: Status 404 returned error can't find the container with id fe163e0b70e3d7259ed9945f97305662bc4fab3f4370d9942e0a1fe7e831d834 Nov 03 20:23:39.814523 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:39.814433 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq" event={"ID":"4ff994a2-65e2-44da-8274-46384f9b1971","Type":"ContainerStarted","Data":"fe163e0b70e3d7259ed9945f97305662bc4fab3f4370d9942e0a1fe7e831d834"} Nov 03 20:23:39.902987 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:39.902943 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 03 20:23:39.903496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:39.903012 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" podUID="28da3b98-e5f9-4355-876f-560f633061de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 03 20:23:40.842618 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:40.842565 5111 generic.go:356] "Generic (PLEG): container finished" podID="4ff994a2-65e2-44da-8274-46384f9b1971" containerID="e9fe4cb7404c6381223617692836d1ea11e1d435220e365b629aa9e48927e0a9" exitCode=0 Nov 03 20:23:40.842851 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:40.842654 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq" event={"ID":"4ff994a2-65e2-44da-8274-46384f9b1971","Type":"ContainerDied","Data":"e9fe4cb7404c6381223617692836d1ea11e1d435220e365b629aa9e48927e0a9"} Nov 03 20:23:40.901854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:40.901813 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 03 20:23:40.902528 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:40.901870 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" podUID="28da3b98-e5f9-4355-876f-560f633061de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 03 20:23:41.902564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:41.902510 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 03 20:23:41.903110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:41.902585 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" podUID="28da3b98-e5f9-4355-876f-560f633061de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 03 20:23:42.165832 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:42.165513 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq" Nov 03 20:23:42.240556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:42.240514 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ff994a2-65e2-44da-8274-46384f9b1971-kube-api-access-dmrz6\" (UniqueName: \"kubernetes.io/projected/4ff994a2-65e2-44da-8274-46384f9b1971-kube-api-access-dmrz6\") pod \"4ff994a2-65e2-44da-8274-46384f9b1971\" (UID: \"4ff994a2-65e2-44da-8274-46384f9b1971\") " Nov 03 20:23:42.240770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:42.240651 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ff994a2-65e2-44da-8274-46384f9b1971-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff994a2-65e2-44da-8274-46384f9b1971-maas-api-openshift-ca\") pod \"4ff994a2-65e2-44da-8274-46384f9b1971\" (UID: \"4ff994a2-65e2-44da-8274-46384f9b1971\") " Nov 03 20:23:42.240845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:42.240777 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ff994a2-65e2-44da-8274-46384f9b1971-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff994a2-65e2-44da-8274-46384f9b1971-maas-api-xks-ca\") pod \"4ff994a2-65e2-44da-8274-46384f9b1971\" (UID: \"4ff994a2-65e2-44da-8274-46384f9b1971\") " Nov 03 20:23:42.241364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:42.241331 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ff994a2-65e2-44da-8274-46384f9b1971-maas-api-openshift-ca" pod "4ff994a2-65e2-44da-8274-46384f9b1971" (UID: "4ff994a2-65e2-44da-8274-46384f9b1971"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:42.241701 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:42.241679 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff994a2-65e2-44da-8274-46384f9b1971-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:42.243013 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:42.242985 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff994a2-65e2-44da-8274-46384f9b1971-kube-api-access-dmrz6" pod "4ff994a2-65e2-44da-8274-46384f9b1971" (UID: "4ff994a2-65e2-44da-8274-46384f9b1971"). InnerVolumeSpecName "kube-api-access-dmrz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:42.255054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:42.255025 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ff994a2-65e2-44da-8274-46384f9b1971-maas-api-xks-ca" pod "4ff994a2-65e2-44da-8274-46384f9b1971" (UID: "4ff994a2-65e2-44da-8274-46384f9b1971"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:42.343574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:42.343491 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmrz6\" (UniqueName: \"kubernetes.io/projected/4ff994a2-65e2-44da-8274-46384f9b1971-kube-api-access-dmrz6\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:42.343574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:42.343529 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff994a2-65e2-44da-8274-46384f9b1971-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:42.896589 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:42.894968 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq" Nov 03 20:23:42.896589 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:42.894999 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq" event={"ID":"4ff994a2-65e2-44da-8274-46384f9b1971","Type":"ContainerDied","Data":"fe163e0b70e3d7259ed9945f97305662bc4fab3f4370d9942e0a1fe7e831d834"} Nov 03 20:23:42.896589 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:42.895067 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe163e0b70e3d7259ed9945f97305662bc4fab3f4370d9942e0a1fe7e831d834" Nov 03 20:23:42.909005 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:42.908968 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:42.909836 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:42.909812 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:42.910713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:42.910691 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:23:43.631556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:43.627215 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784"] Nov 03 20:23:48.990522 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:48.990488 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc"] Nov 03 20:23:49.001648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:49.001613 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc"] Nov 03 20:23:53.992043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:53.992000 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" podUID="97dad6bc-5f5e-49cf-9fde-1470e77512f0" containerName="payload-processing" containerID="cri-o://fc02b840bf54c11f0cc5cfaf4ca0dda12e7c5bf529c908a7311ad637e8f84ac8" gracePeriod=25 Nov 03 20:23:54.002406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.002340 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" podUID="105f9648-6b47-477c-afb1-a41d07095e1b" containerName="payload-pre-processing" containerID="cri-o://9642ceac49c2e7bf0ff1e5dd96112fa9ce84a4c6e7dc79dbe75cff787f3f7dd7" gracePeriod=25 Nov 03 20:23:54.401318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.401279 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc"] Nov 03 20:23:54.402135 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.402117 5111 cpu_manager.go:414] "containerMap: removing container" podUID="4ff994a2-65e2-44da-8274-46384f9b1971" containerName="revoke-keys" Nov 03 20:23:54.402135 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.402135 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ff994a2-65e2-44da-8274-46384f9b1971" containerName="revoke-keys" Nov 03 20:23:54.402293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.402283 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ff994a2-65e2-44da-8274-46384f9b1971" containerName="revoke-keys" Nov 03 20:23:54.407088 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.407063 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" Nov 03 20:23:54.415960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.415663 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:23:54.415960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.415724 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2"] Nov 03 20:23:54.417426 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.417401 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:23:54.426836 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.421768 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" Nov 03 20:23:54.426836 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.426576 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc"] Nov 03 20:23:54.436044 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.436006 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2"] Nov 03 20:23:54.542409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.542369 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjhwm\" (UniqueName: \"kubernetes.io/projected/fbe4e11b-d14b-4b23-83d4-6815212acbed-kube-api-access-jjhwm\") pod \"maas-api-revoke-sub-364c1787-c2tm2\" (UID: \"fbe4e11b-d14b-4b23-83d4-6815212acbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" Nov 03 20:23:54.542623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.542429 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe4e11b-d14b-4b23-83d4-6815212acbed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-364c1787-c2tm2\" (UID: \"fbe4e11b-d14b-4b23-83d4-6815212acbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" Nov 03 20:23:54.542623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.542490 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe4e11b-d14b-4b23-83d4-6815212acbed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-364c1787-c2tm2\" (UID: \"fbe4e11b-d14b-4b23-83d4-6815212acbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" Nov 03 20:23:54.542745 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.542701 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebbac93-2f53-40cb-904f-d8a25729e213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22f7ff14-n5hmc\" (UID: \"1ebbac93-2f53-40cb-904f-d8a25729e213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" Nov 03 20:23:54.542745 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.542738 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxm89\" (UniqueName: \"kubernetes.io/projected/1ebbac93-2f53-40cb-904f-d8a25729e213-kube-api-access-pxm89\") pod \"maas-api-revoke-sub-22f7ff14-n5hmc\" (UID: \"1ebbac93-2f53-40cb-904f-d8a25729e213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" Nov 03 20:23:54.542938 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.542922 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebbac93-2f53-40cb-904f-d8a25729e213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22f7ff14-n5hmc\" (UID: \"1ebbac93-2f53-40cb-904f-d8a25729e213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" Nov 03 20:23:54.644877 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.644840 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebbac93-2f53-40cb-904f-d8a25729e213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22f7ff14-n5hmc\" (UID: \"1ebbac93-2f53-40cb-904f-d8a25729e213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" Nov 03 20:23:54.645094 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.644948 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjhwm\" (UniqueName: \"kubernetes.io/projected/fbe4e11b-d14b-4b23-83d4-6815212acbed-kube-api-access-jjhwm\") pod \"maas-api-revoke-sub-364c1787-c2tm2\" (UID: \"fbe4e11b-d14b-4b23-83d4-6815212acbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" Nov 03 20:23:54.645094 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.644985 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe4e11b-d14b-4b23-83d4-6815212acbed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-364c1787-c2tm2\" (UID: \"fbe4e11b-d14b-4b23-83d4-6815212acbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" Nov 03 20:23:54.645094 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.645012 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe4e11b-d14b-4b23-83d4-6815212acbed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-364c1787-c2tm2\" (UID: \"fbe4e11b-d14b-4b23-83d4-6815212acbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" Nov 03 20:23:54.645271 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.645131 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebbac93-2f53-40cb-904f-d8a25729e213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22f7ff14-n5hmc\" (UID: \"1ebbac93-2f53-40cb-904f-d8a25729e213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" Nov 03 20:23:54.645271 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.645175 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxm89\" (UniqueName: \"kubernetes.io/projected/1ebbac93-2f53-40cb-904f-d8a25729e213-kube-api-access-pxm89\") pod \"maas-api-revoke-sub-22f7ff14-n5hmc\" (UID: \"1ebbac93-2f53-40cb-904f-d8a25729e213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" Nov 03 20:23:54.646340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.646298 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebbac93-2f53-40cb-904f-d8a25729e213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22f7ff14-n5hmc\" (UID: \"1ebbac93-2f53-40cb-904f-d8a25729e213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" Nov 03 20:23:54.646484 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.646432 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe4e11b-d14b-4b23-83d4-6815212acbed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-364c1787-c2tm2\" (UID: \"fbe4e11b-d14b-4b23-83d4-6815212acbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" Nov 03 20:23:54.647889 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.647860 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebbac93-2f53-40cb-904f-d8a25729e213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22f7ff14-n5hmc\" (UID: \"1ebbac93-2f53-40cb-904f-d8a25729e213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" Nov 03 20:23:54.647979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.647942 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe4e11b-d14b-4b23-83d4-6815212acbed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-364c1787-c2tm2\" (UID: \"fbe4e11b-d14b-4b23-83d4-6815212acbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" Nov 03 20:23:54.654109 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.654046 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxm89\" (UniqueName: \"kubernetes.io/projected/1ebbac93-2f53-40cb-904f-d8a25729e213-kube-api-access-pxm89\") pod \"maas-api-revoke-sub-22f7ff14-n5hmc\" (UID: \"1ebbac93-2f53-40cb-904f-d8a25729e213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" Nov 03 20:23:54.654207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.654121 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjhwm\" (UniqueName: \"kubernetes.io/projected/fbe4e11b-d14b-4b23-83d4-6815212acbed-kube-api-access-jjhwm\") pod \"maas-api-revoke-sub-364c1787-c2tm2\" (UID: \"fbe4e11b-d14b-4b23-83d4-6815212acbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" Nov 03 20:23:54.753395 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.753357 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" Nov 03 20:23:54.766791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:54.766757 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" Nov 03 20:23:55.087647 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:55.087603 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" podUID="105f9648-6b47-477c-afb1-a41d07095e1b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:23:55.088122 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:55.087992 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" podUID="97dad6bc-5f5e-49cf-9fde-1470e77512f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:23:55.121152 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:55.121117 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2"] Nov 03 20:23:55.121509 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:23:55.121482 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe4e11b_d14b_4b23_83d4_6815212acbed.slice/crio-06924a41401ab6ce3f9507109810b00d8ac963ae8925617984be2b6769e98169 WatchSource:0}: Error finding container 06924a41401ab6ce3f9507109810b00d8ac963ae8925617984be2b6769e98169: Status 404 returned error can't find the container with id 06924a41401ab6ce3f9507109810b00d8ac963ae8925617984be2b6769e98169 Nov 03 20:23:55.134257 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:55.134232 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc"] Nov 03 20:23:55.137256 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:23:55.137225 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebbac93_2f53_40cb_904f_d8a25729e213.slice/crio-e78431ba2de0775ba8000e8c75c59c1e69275c7f8e40608203ad84a40edd405f WatchSource:0}: Error finding container e78431ba2de0775ba8000e8c75c59c1e69275c7f8e40608203ad84a40edd405f: Status 404 returned error can't find the container with id e78431ba2de0775ba8000e8c75c59c1e69275c7f8e40608203ad84a40edd405f Nov 03 20:23:56.045587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:56.045443 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" event={"ID":"fbe4e11b-d14b-4b23-83d4-6815212acbed","Type":"ContainerStarted","Data":"1ffd3cd0bb7b3127c81162845e22453434dc64ed5f5934ee29e9932ac3f194c2"} Nov 03 20:23:56.045587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:56.045527 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" event={"ID":"fbe4e11b-d14b-4b23-83d4-6815212acbed","Type":"ContainerStarted","Data":"06924a41401ab6ce3f9507109810b00d8ac963ae8925617984be2b6769e98169"} Nov 03 20:23:56.047730 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:56.047703 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" event={"ID":"1ebbac93-2f53-40cb-904f-d8a25729e213","Type":"ContainerStarted","Data":"4fa27356325fad63671b69b771198284207c6d41ab4e2a20ccb38ed21d3e2a9a"} Nov 03 20:23:56.048459 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:56.047748 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" event={"ID":"1ebbac93-2f53-40cb-904f-d8a25729e213","Type":"ContainerStarted","Data":"e78431ba2de0775ba8000e8c75c59c1e69275c7f8e40608203ad84a40edd405f"} Nov 03 20:23:56.066310 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:56.066263 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" podStartSLOduration=2.066226573 podStartE2EDuration="2.066226573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:56.063360979 +0000 UTC m=+2666.025739417" watchObservedRunningTime="2026-11-03 20:23:56.066226573 +0000 UTC m=+2666.028605054" Nov 03 20:23:56.082543 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:56.082485 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" podStartSLOduration=2.082475772 podStartE2EDuration="2.082475772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:56.079896698 +0000 UTC m=+2666.042275136" watchObservedRunningTime="2026-11-03 20:23:56.082475772 +0000 UTC m=+2666.044854209" Nov 03 20:23:57.058714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:57.058675 5111 generic.go:356] "Generic (PLEG): container finished" podID="1ebbac93-2f53-40cb-904f-d8a25729e213" containerID="4fa27356325fad63671b69b771198284207c6d41ab4e2a20ccb38ed21d3e2a9a" exitCode=0 Nov 03 20:23:57.059220 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:57.058750 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" event={"ID":"1ebbac93-2f53-40cb-904f-d8a25729e213","Type":"ContainerDied","Data":"4fa27356325fad63671b69b771198284207c6d41ab4e2a20ccb38ed21d3e2a9a"} Nov 03 20:23:57.060790 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:57.060763 5111 generic.go:356] "Generic (PLEG): container finished" podID="fbe4e11b-d14b-4b23-83d4-6815212acbed" containerID="1ffd3cd0bb7b3127c81162845e22453434dc64ed5f5934ee29e9932ac3f194c2" exitCode=0 Nov 03 20:23:57.060908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:57.060799 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" event={"ID":"fbe4e11b-d14b-4b23-83d4-6815212acbed","Type":"ContainerDied","Data":"1ffd3cd0bb7b3127c81162845e22453434dc64ed5f5934ee29e9932ac3f194c2"} Nov 03 20:23:58.424864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.424836 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" Nov 03 20:23:58.429353 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.429137 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" Nov 03 20:23:58.444482 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.444041 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe4e11b-d14b-4b23-83d4-6815212acbed-kube-api-access-jjhwm\" (UniqueName: \"kubernetes.io/projected/fbe4e11b-d14b-4b23-83d4-6815212acbed-kube-api-access-jjhwm\") pod \"fbe4e11b-d14b-4b23-83d4-6815212acbed\" (UID: \"fbe4e11b-d14b-4b23-83d4-6815212acbed\") " Nov 03 20:23:58.445104 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.444215 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbe4e11b-d14b-4b23-83d4-6815212acbed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe4e11b-d14b-4b23-83d4-6815212acbed-maas-api-xks-ca\") pod \"fbe4e11b-d14b-4b23-83d4-6815212acbed\" (UID: \"fbe4e11b-d14b-4b23-83d4-6815212acbed\") " Nov 03 20:23:58.445241 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.445182 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe4e11b-d14b-4b23-83d4-6815212acbed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe4e11b-d14b-4b23-83d4-6815212acbed-maas-api-openshift-ca\") pod \"fbe4e11b-d14b-4b23-83d4-6815212acbed\" (UID: \"fbe4e11b-d14b-4b23-83d4-6815212acbed\") " Nov 03 20:23:58.445241 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.445225 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ebbac93-2f53-40cb-904f-d8a25729e213-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebbac93-2f53-40cb-904f-d8a25729e213-maas-api-xks-ca\") pod \"1ebbac93-2f53-40cb-904f-d8a25729e213\" (UID: \"1ebbac93-2f53-40cb-904f-d8a25729e213\") " Nov 03 20:23:58.445356 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.445283 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ebbac93-2f53-40cb-904f-d8a25729e213-kube-api-access-pxm89\" (UniqueName: \"kubernetes.io/projected/1ebbac93-2f53-40cb-904f-d8a25729e213-kube-api-access-pxm89\") pod \"1ebbac93-2f53-40cb-904f-d8a25729e213\" (UID: \"1ebbac93-2f53-40cb-904f-d8a25729e213\") " Nov 03 20:23:58.445576 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.445556 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ebbac93-2f53-40cb-904f-d8a25729e213-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebbac93-2f53-40cb-904f-d8a25729e213-maas-api-openshift-ca\") pod \"1ebbac93-2f53-40cb-904f-d8a25729e213\" (UID: \"1ebbac93-2f53-40cb-904f-d8a25729e213\") " Nov 03 20:23:58.446706 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.446653 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe4e11b-d14b-4b23-83d4-6815212acbed-maas-api-openshift-ca" pod "fbe4e11b-d14b-4b23-83d4-6815212acbed" (UID: "fbe4e11b-d14b-4b23-83d4-6815212acbed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:58.447332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.447242 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe4e11b-d14b-4b23-83d4-6815212acbed-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:58.447888 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.447832 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebbac93-2f53-40cb-904f-d8a25729e213-maas-api-openshift-ca" pod "1ebbac93-2f53-40cb-904f-d8a25729e213" (UID: "1ebbac93-2f53-40cb-904f-d8a25729e213"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:58.448400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.448279 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbe4e11b-d14b-4b23-83d4-6815212acbed-maas-api-xks-ca" pod "fbe4e11b-d14b-4b23-83d4-6815212acbed" (UID: "fbe4e11b-d14b-4b23-83d4-6815212acbed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:58.448400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.448340 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe4e11b-d14b-4b23-83d4-6815212acbed-kube-api-access-jjhwm" pod "fbe4e11b-d14b-4b23-83d4-6815212acbed" (UID: "fbe4e11b-d14b-4b23-83d4-6815212acbed"). InnerVolumeSpecName "kube-api-access-jjhwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:58.449974 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.449948 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebbac93-2f53-40cb-904f-d8a25729e213-kube-api-access-pxm89" pod "1ebbac93-2f53-40cb-904f-d8a25729e213" (UID: "1ebbac93-2f53-40cb-904f-d8a25729e213"). InnerVolumeSpecName "kube-api-access-pxm89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:58.451412 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.451382 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebbac93-2f53-40cb-904f-d8a25729e213-maas-api-xks-ca" pod "1ebbac93-2f53-40cb-904f-d8a25729e213" (UID: "1ebbac93-2f53-40cb-904f-d8a25729e213"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:58.555600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.549890 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebbac93-2f53-40cb-904f-d8a25729e213-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:58.555600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.549929 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjhwm\" (UniqueName: \"kubernetes.io/projected/fbe4e11b-d14b-4b23-83d4-6815212acbed-kube-api-access-jjhwm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:58.555600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.549956 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe4e11b-d14b-4b23-83d4-6815212acbed-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:58.555600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.549971 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebbac93-2f53-40cb-904f-d8a25729e213-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:58.555600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.549985 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxm89\" (UniqueName: \"kubernetes.io/projected/1ebbac93-2f53-40cb-904f-d8a25729e213-kube-api-access-pxm89\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:58.674833 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:58.674773 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" containerID="cri-o://e1403371711eec094adf68f6d9c62e935510394c6bb9b4e37d522b12c7c1ba64" gracePeriod=45 Nov 03 20:23:59.028079 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.028053 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:23:59.066553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.065224 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5333c660-c55d-40f0-b0cc-8decef5eeb22-tls-certs\" (UniqueName: \"kubernetes.io/secret/5333c660-c55d-40f0-b0cc-8decef5eeb22-tls-certs\") pod \"5333c660-c55d-40f0-b0cc-8decef5eeb22\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " Nov 03 20:23:59.066553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.065280 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-tmp-dir\") pod \"5333c660-c55d-40f0-b0cc-8decef5eeb22\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " Nov 03 20:23:59.066553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.065357 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-home\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-home\") pod \"5333c660-c55d-40f0-b0cc-8decef5eeb22\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " Nov 03 20:23:59.066553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.065457 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-model-cache\") pod \"5333c660-c55d-40f0-b0cc-8decef5eeb22\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " Nov 03 20:23:59.066553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.065623 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5333c660-c55d-40f0-b0cc-8decef5eeb22-kube-api-access-9xs79\" (UniqueName: \"kubernetes.io/projected/5333c660-c55d-40f0-b0cc-8decef5eeb22-kube-api-access-9xs79\") pod \"5333c660-c55d-40f0-b0cc-8decef5eeb22\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " Nov 03 20:23:59.066553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.065670 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-dshm\") pod \"5333c660-c55d-40f0-b0cc-8decef5eeb22\" (UID: \"5333c660-c55d-40f0-b0cc-8decef5eeb22\") " Nov 03 20:23:59.067012 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.066962 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-home" pod "5333c660-c55d-40f0-b0cc-8decef5eeb22" (UID: "5333c660-c55d-40f0-b0cc-8decef5eeb22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:59.067230 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.067189 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-tmp-dir" pod "5333c660-c55d-40f0-b0cc-8decef5eeb22" (UID: "5333c660-c55d-40f0-b0cc-8decef5eeb22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:59.067812 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.067781 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-model-cache" pod "5333c660-c55d-40f0-b0cc-8decef5eeb22" (UID: "5333c660-c55d-40f0-b0cc-8decef5eeb22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:59.069395 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.069345 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-dshm" pod "5333c660-c55d-40f0-b0cc-8decef5eeb22" (UID: "5333c660-c55d-40f0-b0cc-8decef5eeb22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:59.069505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.069426 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5333c660-c55d-40f0-b0cc-8decef5eeb22-tls-certs" pod "5333c660-c55d-40f0-b0cc-8decef5eeb22" (UID: "5333c660-c55d-40f0-b0cc-8decef5eeb22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:59.070845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.070823 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5333c660-c55d-40f0-b0cc-8decef5eeb22-kube-api-access-9xs79" pod "5333c660-c55d-40f0-b0cc-8decef5eeb22" (UID: "5333c660-c55d-40f0-b0cc-8decef5eeb22"). InnerVolumeSpecName "kube-api-access-9xs79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:59.081693 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.081672 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" Nov 03 20:23:59.081800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.081760 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc" event={"ID":"1ebbac93-2f53-40cb-904f-d8a25729e213","Type":"ContainerDied","Data":"e78431ba2de0775ba8000e8c75c59c1e69275c7f8e40608203ad84a40edd405f"} Nov 03 20:23:59.081853 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.081827 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e78431ba2de0775ba8000e8c75c59c1e69275c7f8e40608203ad84a40edd405f" Nov 03 20:23:59.086745 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.084096 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" event={"ID":"fbe4e11b-d14b-4b23-83d4-6815212acbed","Type":"ContainerDied","Data":"06924a41401ab6ce3f9507109810b00d8ac963ae8925617984be2b6769e98169"} Nov 03 20:23:59.086745 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.084142 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06924a41401ab6ce3f9507109810b00d8ac963ae8925617984be2b6769e98169" Nov 03 20:23:59.086745 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.084104 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2" Nov 03 20:23:59.086745 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.085907 5111 generic.go:356] "Generic (PLEG): container finished" podID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerID="e1403371711eec094adf68f6d9c62e935510394c6bb9b4e37d522b12c7c1ba64" exitCode=0 Nov 03 20:23:59.086745 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.085985 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" event={"ID":"5333c660-c55d-40f0-b0cc-8decef5eeb22","Type":"ContainerDied","Data":"e1403371711eec094adf68f6d9c62e935510394c6bb9b4e37d522b12c7c1ba64"} Nov 03 20:23:59.086745 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.086017 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" event={"ID":"5333c660-c55d-40f0-b0cc-8decef5eeb22","Type":"ContainerDied","Data":"51a4b73281b1d5ff3a82348167bc2d647fe6c220737e2bbb759d36b62e5a5fa0"} Nov 03 20:23:59.086745 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.086039 5111 scope.go:122] "RemoveContainer" containerID="e1403371711eec094adf68f6d9c62e935510394c6bb9b4e37d522b12c7c1ba64" Nov 03 20:23:59.086745 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.086197 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" Nov 03 20:23:59.117357 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.117322 5111 scope.go:122] "RemoveContainer" containerID="e1403371711eec094adf68f6d9c62e935510394c6bb9b4e37d522b12c7c1ba64" Nov 03 20:23:59.117741 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:23:59.117717 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1403371711eec094adf68f6d9c62e935510394c6bb9b4e37d522b12c7c1ba64\": container with ID starting with e1403371711eec094adf68f6d9c62e935510394c6bb9b4e37d522b12c7c1ba64 not found: ID does not exist" containerID="e1403371711eec094adf68f6d9c62e935510394c6bb9b4e37d522b12c7c1ba64" Nov 03 20:23:59.117876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.117753 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1403371711eec094adf68f6d9c62e935510394c6bb9b4e37d522b12c7c1ba64"} err="failed to get container status \"e1403371711eec094adf68f6d9c62e935510394c6bb9b4e37d522b12c7c1ba64\": rpc error: code = NotFound desc = could not find container \"e1403371711eec094adf68f6d9c62e935510394c6bb9b4e37d522b12c7c1ba64\": container with ID starting with e1403371711eec094adf68f6d9c62e935510394c6bb9b4e37d522b12c7c1ba64 not found: ID does not exist" Nov 03 20:23:59.128787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.128745 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784"] Nov 03 20:23:59.134450 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.134417 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784"] Nov 03 20:23:59.167930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.167893 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xs79\" (UniqueName: \"kubernetes.io/projected/5333c660-c55d-40f0-b0cc-8decef5eeb22-kube-api-access-9xs79\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:59.167930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.167925 5111 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-dshm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:59.167930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.167935 5111 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5333c660-c55d-40f0-b0cc-8decef5eeb22-tls-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:59.168177 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.167944 5111 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-tmp-dir\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:59.168177 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.167957 5111 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-home\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:59.168177 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.167969 5111 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5333c660-c55d-40f0-b0cc-8decef5eeb22-model-cache\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:59.972696 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:23:59.972657 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/premium-w3-b87733-kserve-54b489bc6f-bk784" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:00.187456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:00.187423 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" path="/var/lib/kubelet/pods/5333c660-c55d-40f0-b0cc-8decef5eeb22/volumes" Nov 03 20:24:05.064503 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:24:05.064468 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:24:05.087008 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:05.086972 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" podUID="105f9648-6b47-477c-afb1-a41d07095e1b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:24:05.087407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:05.087374 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" podUID="97dad6bc-5f5e-49cf-9fde-1470e77512f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:24:11.738704 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:11.738669 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq"] Nov 03 20:24:11.745470 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:11.745440 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b87733-n54vq"] Nov 03 20:24:12.186236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:12.186198 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ff994a2-65e2-44da-8274-46384f9b1971" path="/var/lib/kubelet/pods/4ff994a2-65e2-44da-8274-46384f9b1971/volumes" Nov 03 20:24:14.224466 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.224435 5111 generic.go:356] "Generic (PLEG): container finished" podID="105f9648-6b47-477c-afb1-a41d07095e1b" containerID="9642ceac49c2e7bf0ff1e5dd96112fa9ce84a4c6e7dc79dbe75cff787f3f7dd7" exitCode=0 Nov 03 20:24:14.224932 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.224523 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" event={"ID":"105f9648-6b47-477c-afb1-a41d07095e1b","Type":"ContainerDied","Data":"9642ceac49c2e7bf0ff1e5dd96112fa9ce84a4c6e7dc79dbe75cff787f3f7dd7"} Nov 03 20:24:14.226901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.226866 5111 generic.go:356] "Generic (PLEG): container finished" podID="97dad6bc-5f5e-49cf-9fde-1470e77512f0" containerID="fc02b840bf54c11f0cc5cfaf4ca0dda12e7c5bf529c908a7311ad637e8f84ac8" exitCode=0 Nov 03 20:24:14.227058 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.226933 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" event={"ID":"97dad6bc-5f5e-49cf-9fde-1470e77512f0","Type":"ContainerDied","Data":"fc02b840bf54c11f0cc5cfaf4ca0dda12e7c5bf529c908a7311ad637e8f84ac8"} Nov 03 20:24:14.377989 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.377962 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" Nov 03 20:24:14.425048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.425016 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" Nov 03 20:24:14.478423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.472930 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97dad6bc-5f5e-49cf-9fde-1470e77512f0-kube-api-access-xn88p\" (UniqueName: \"kubernetes.io/projected/97dad6bc-5f5e-49cf-9fde-1470e77512f0-kube-api-access-xn88p\") pod \"97dad6bc-5f5e-49cf-9fde-1470e77512f0\" (UID: \"97dad6bc-5f5e-49cf-9fde-1470e77512f0\") " Nov 03 20:24:14.478423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.472996 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105f9648-6b47-477c-afb1-a41d07095e1b-kube-api-access-pcv22\" (UniqueName: \"kubernetes.io/projected/105f9648-6b47-477c-afb1-a41d07095e1b-kube-api-access-pcv22\") pod \"105f9648-6b47-477c-afb1-a41d07095e1b\" (UID: \"105f9648-6b47-477c-afb1-a41d07095e1b\") " Nov 03 20:24:14.478423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.473204 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/105f9648-6b47-477c-afb1-a41d07095e1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105f9648-6b47-477c-afb1-a41d07095e1b-plugins-config-volume\") pod \"105f9648-6b47-477c-afb1-a41d07095e1b\" (UID: \"105f9648-6b47-477c-afb1-a41d07095e1b\") " Nov 03 20:24:14.478423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.473234 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97dad6bc-5f5e-49cf-9fde-1470e77512f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97dad6bc-5f5e-49cf-9fde-1470e77512f0-plugins-config-volume\") pod \"97dad6bc-5f5e-49cf-9fde-1470e77512f0\" (UID: \"97dad6bc-5f5e-49cf-9fde-1470e77512f0\") " Nov 03 20:24:14.478423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.476435 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97dad6bc-5f5e-49cf-9fde-1470e77512f0-plugins-config-volume" pod "97dad6bc-5f5e-49cf-9fde-1470e77512f0" (UID: "97dad6bc-5f5e-49cf-9fde-1470e77512f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:14.478423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.477657 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105f9648-6b47-477c-afb1-a41d07095e1b-plugins-config-volume" pod "105f9648-6b47-477c-afb1-a41d07095e1b" (UID: "105f9648-6b47-477c-afb1-a41d07095e1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:14.480430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.480397 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105f9648-6b47-477c-afb1-a41d07095e1b-kube-api-access-pcv22" pod "105f9648-6b47-477c-afb1-a41d07095e1b" (UID: "105f9648-6b47-477c-afb1-a41d07095e1b"). InnerVolumeSpecName "kube-api-access-pcv22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:14.481896 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.481863 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97dad6bc-5f5e-49cf-9fde-1470e77512f0-kube-api-access-xn88p" pod "97dad6bc-5f5e-49cf-9fde-1470e77512f0" (UID: "97dad6bc-5f5e-49cf-9fde-1470e77512f0"). InnerVolumeSpecName "kube-api-access-xn88p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:14.575438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.575347 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn88p\" (UniqueName: \"kubernetes.io/projected/97dad6bc-5f5e-49cf-9fde-1470e77512f0-kube-api-access-xn88p\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:14.575438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.575387 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcv22\" (UniqueName: \"kubernetes.io/projected/105f9648-6b47-477c-afb1-a41d07095e1b-kube-api-access-pcv22\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:14.575438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.575397 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105f9648-6b47-477c-afb1-a41d07095e1b-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:14.575438 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:14.575407 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97dad6bc-5f5e-49cf-9fde-1470e77512f0-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:15.237167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:15.237133 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" Nov 03 20:24:15.237665 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:15.237133 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc" event={"ID":"105f9648-6b47-477c-afb1-a41d07095e1b","Type":"ContainerDied","Data":"a3ce06a49d757ea93f704c2792784b259f23ceee6e7dabead092e689ca10efbd"} Nov 03 20:24:15.237665 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:15.237267 5111 scope.go:122] "RemoveContainer" containerID="9642ceac49c2e7bf0ff1e5dd96112fa9ce84a4c6e7dc79dbe75cff787f3f7dd7" Nov 03 20:24:15.238893 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:15.238842 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" event={"ID":"97dad6bc-5f5e-49cf-9fde-1470e77512f0","Type":"ContainerDied","Data":"d1932f73a82dea91cacab9069e70dd7d4b9bb1463c0455773417365c9f107b40"} Nov 03 20:24:15.238893 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:15.238863 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc" Nov 03 20:24:15.249613 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:15.249590 5111 scope.go:122] "RemoveContainer" containerID="fc02b840bf54c11f0cc5cfaf4ca0dda12e7c5bf529c908a7311ad637e8f84ac8" Nov 03 20:24:15.266661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:15.266568 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc"] Nov 03 20:24:15.271257 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:15.271227 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b87733-9c5b7fdd-d4bsc"] Nov 03 20:24:15.281044 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:15.281016 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc"] Nov 03 20:24:15.289552 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:15.289326 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b87733-747d96db47-wv2pc"] Nov 03 20:24:16.202174 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:16.202140 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="105f9648-6b47-477c-afb1-a41d07095e1b" path="/var/lib/kubelet/pods/105f9648-6b47-477c-afb1-a41d07095e1b/volumes" Nov 03 20:24:16.203369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:16.203340 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97dad6bc-5f5e-49cf-9fde-1470e77512f0" path="/var/lib/kubelet/pods/97dad6bc-5f5e-49cf-9fde-1470e77512f0/volumes" Nov 03 20:24:19.323778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:19.322869 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w"] Nov 03 20:24:19.323778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:19.323115 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" podUID="8bf7d094-ba6e-4c14-b731-c13068585df1" containerName="istio-proxy" containerID="cri-o://d0a9bbe12d9d63b469c03867a5799429153313f12e9ab36c577f56a29f4724fc" gracePeriod=30 Nov 03 20:24:19.372712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:19.371705 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2"] Nov 03 20:24:19.372712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:19.371958 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" podUID="c1f87478-ca01-41de-b602-f5a831e07913" containerName="istio-proxy" containerID="cri-o://6deebfa3bdcf1565192a7b61b9474c838a90a90004bb0239c66e59bebb122765" gracePeriod=30 Nov 03 20:24:24.784555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.783107 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:24:24.837582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.837402 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:24:24.933723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.933684 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-workload-socket\") pod \"c1f87478-ca01-41de-b602-f5a831e07913\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " Nov 03 20:24:24.933981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.933746 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-workload-socket\") pod \"8bf7d094-ba6e-4c14-b731-c13068585df1\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " Nov 03 20:24:24.933981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.933781 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf7d094-ba6e-4c14-b731-c13068585df1-kube-api-access-n4n2f\" (UniqueName: \"kubernetes.io/projected/8bf7d094-ba6e-4c14-b731-c13068585df1-kube-api-access-n4n2f\") pod \"8bf7d094-ba6e-4c14-b731-c13068585df1\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " Nov 03 20:24:24.933981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.933810 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-envoy\") pod \"8bf7d094-ba6e-4c14-b731-c13068585df1\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " Nov 03 20:24:24.933981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.933906 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-istio-data\") pod \"c1f87478-ca01-41de-b602-f5a831e07913\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " Nov 03 20:24:24.933981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.933961 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-workload-certs\") pod \"8bf7d094-ba6e-4c14-b731-c13068585df1\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " Nov 03 20:24:24.934279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.933988 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-data\") pod \"8bf7d094-ba6e-4c14-b731-c13068585df1\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " Nov 03 20:24:24.934279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.934036 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-workload-socket" pod "c1f87478-ca01-41de-b602-f5a831e07913" (UID: "c1f87478-ca01-41de-b602-f5a831e07913"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:24.934279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.934050 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1f87478-ca01-41de-b602-f5a831e07913-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f87478-ca01-41de-b602-f5a831e07913-istio-podinfo\") pod \"c1f87478-ca01-41de-b602-f5a831e07913\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " Nov 03 20:24:24.934279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.934148 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-podinfo\") pod \"8bf7d094-ba6e-4c14-b731-c13068585df1\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " Nov 03 20:24:24.934279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.934263 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-workload-certs\") pod \"c1f87478-ca01-41de-b602-f5a831e07913\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " Nov 03 20:24:24.934524 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.934322 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-token\" (UniqueName: \"kubernetes.io/projected/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-token\") pod \"8bf7d094-ba6e-4c14-b731-c13068585df1\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " Nov 03 20:24:24.934524 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.934392 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-credential-socket\") pod \"8bf7d094-ba6e-4c14-b731-c13068585df1\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " Nov 03 20:24:24.934524 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.934429 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf7d094-ba6e-4c14-b731-c13068585df1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf7d094-ba6e-4c14-b731-c13068585df1-istiod-ca-cert\") pod \"8bf7d094-ba6e-4c14-b731-c13068585df1\" (UID: \"8bf7d094-ba6e-4c14-b731-c13068585df1\") " Nov 03 20:24:24.934524 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.934471 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f87478-ca01-41de-b602-f5a831e07913-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f87478-ca01-41de-b602-f5a831e07913-istiod-ca-cert\") pod \"c1f87478-ca01-41de-b602-f5a831e07913\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " Nov 03 20:24:24.934791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.934554 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-istio-envoy\") pod \"c1f87478-ca01-41de-b602-f5a831e07913\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " Nov 03 20:24:24.934791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.934647 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f87478-ca01-41de-b602-f5a831e07913-istio-token\" (UniqueName: \"kubernetes.io/projected/c1f87478-ca01-41de-b602-f5a831e07913-istio-token\") pod \"c1f87478-ca01-41de-b602-f5a831e07913\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " Nov 03 20:24:24.934791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.934703 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-credential-socket\") pod \"c1f87478-ca01-41de-b602-f5a831e07913\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " Nov 03 20:24:24.934791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.934778 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f87478-ca01-41de-b602-f5a831e07913-kube-api-access-g9lpx\" (UniqueName: \"kubernetes.io/projected/c1f87478-ca01-41de-b602-f5a831e07913-kube-api-access-g9lpx\") pod \"c1f87478-ca01-41de-b602-f5a831e07913\" (UID: \"c1f87478-ca01-41de-b602-f5a831e07913\") " Nov 03 20:24:24.935624 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.935601 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:24.936448 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.936416 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-data" pod "8bf7d094-ba6e-4c14-b731-c13068585df1" (UID: "8bf7d094-ba6e-4c14-b731-c13068585df1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:24.936831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.936678 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-workload-certs" pod "8bf7d094-ba6e-4c14-b731-c13068585df1" (UID: "8bf7d094-ba6e-4c14-b731-c13068585df1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:24.936831 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.936739 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1f87478-ca01-41de-b602-f5a831e07913-istio-podinfo" pod "c1f87478-ca01-41de-b602-f5a831e07913" (UID: "c1f87478-ca01-41de-b602-f5a831e07913"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:24:24.937069 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.937048 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf7d094-ba6e-4c14-b731-c13068585df1-istiod-ca-cert" pod "8bf7d094-ba6e-4c14-b731-c13068585df1" (UID: "8bf7d094-ba6e-4c14-b731-c13068585df1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:24.937069 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.937057 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-istio-data" pod "c1f87478-ca01-41de-b602-f5a831e07913" (UID: "c1f87478-ca01-41de-b602-f5a831e07913"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:24.937408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.937386 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-workload-socket" pod "8bf7d094-ba6e-4c14-b731-c13068585df1" (UID: "8bf7d094-ba6e-4c14-b731-c13068585df1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:24.938245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.938020 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-workload-certs" pod "c1f87478-ca01-41de-b602-f5a831e07913" (UID: "c1f87478-ca01-41de-b602-f5a831e07913"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:24.938245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.938167 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f87478-ca01-41de-b602-f5a831e07913-kube-api-access-g9lpx" pod "c1f87478-ca01-41de-b602-f5a831e07913" (UID: "c1f87478-ca01-41de-b602-f5a831e07913"). InnerVolumeSpecName "kube-api-access-g9lpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:24.938810 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.938780 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f87478-ca01-41de-b602-f5a831e07913-istiod-ca-cert" pod "c1f87478-ca01-41de-b602-f5a831e07913" (UID: "c1f87478-ca01-41de-b602-f5a831e07913"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:24.939101 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.939076 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-credential-socket" pod "c1f87478-ca01-41de-b602-f5a831e07913" (UID: "c1f87478-ca01-41de-b602-f5a831e07913"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:24.939323 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.939300 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-credential-socket" pod "8bf7d094-ba6e-4c14-b731-c13068585df1" (UID: "8bf7d094-ba6e-4c14-b731-c13068585df1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:24.939723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.939677 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf7d094-ba6e-4c14-b731-c13068585df1-kube-api-access-n4n2f" pod "8bf7d094-ba6e-4c14-b731-c13068585df1" (UID: "8bf7d094-ba6e-4c14-b731-c13068585df1"). InnerVolumeSpecName "kube-api-access-n4n2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:24.940026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.940003 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-token" pod "8bf7d094-ba6e-4c14-b731-c13068585df1" (UID: "8bf7d094-ba6e-4c14-b731-c13068585df1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:24.940206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.940185 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-podinfo" pod "8bf7d094-ba6e-4c14-b731-c13068585df1" (UID: "8bf7d094-ba6e-4c14-b731-c13068585df1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:24:24.940792 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.940768 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-envoy" pod "8bf7d094-ba6e-4c14-b731-c13068585df1" (UID: "8bf7d094-ba6e-4c14-b731-c13068585df1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:24.940876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.940802 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f87478-ca01-41de-b602-f5a831e07913-istio-token" pod "c1f87478-ca01-41de-b602-f5a831e07913" (UID: "c1f87478-ca01-41de-b602-f5a831e07913"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:24.941893 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:24.941869 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-istio-envoy" pod "c1f87478-ca01-41de-b602-f5a831e07913" (UID: "c1f87478-ca01-41de-b602-f5a831e07913"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:25.037102 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037002 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.037102 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037058 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f87478-ca01-41de-b602-f5a831e07913-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.037102 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037074 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.037102 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037090 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9lpx\" (UniqueName: \"kubernetes.io/projected/c1f87478-ca01-41de-b602-f5a831e07913-kube-api-access-g9lpx\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.037102 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037105 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.037487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037120 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4n2f\" (UniqueName: \"kubernetes.io/projected/8bf7d094-ba6e-4c14-b731-c13068585df1-kube-api-access-n4n2f\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.037487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037134 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.037487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037147 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.037487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037160 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.037487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037174 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.037487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037186 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f87478-ca01-41de-b602-f5a831e07913-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.037487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037199 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.037487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037213 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f87478-ca01-41de-b602-f5a831e07913-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.037487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037226 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf7d094-ba6e-4c14-b731-c13068585df1-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.037487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037239 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf7d094-ba6e-4c14-b731-c13068585df1-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.037487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037252 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf7d094-ba6e-4c14-b731-c13068585df1-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.037487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.037267 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f87478-ca01-41de-b602-f5a831e07913-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:25.394521 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.394487 5111 generic.go:356] "Generic (PLEG): container finished" podID="c1f87478-ca01-41de-b602-f5a831e07913" containerID="6deebfa3bdcf1565192a7b61b9474c838a90a90004bb0239c66e59bebb122765" exitCode=0 Nov 03 20:24:25.394743 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.394695 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" event={"ID":"c1f87478-ca01-41de-b602-f5a831e07913","Type":"ContainerDied","Data":"6deebfa3bdcf1565192a7b61b9474c838a90a90004bb0239c66e59bebb122765"} Nov 03 20:24:25.394743 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.394719 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" Nov 03 20:24:25.394856 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.394750 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2" event={"ID":"c1f87478-ca01-41de-b602-f5a831e07913","Type":"ContainerDied","Data":"51933282ba8d81b2eb68a274cfde3d157721a637069736024df1e28a7bce9392"} Nov 03 20:24:25.394856 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.394777 5111 scope.go:122] "RemoveContainer" containerID="6deebfa3bdcf1565192a7b61b9474c838a90a90004bb0239c66e59bebb122765" Nov 03 20:24:25.396360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.396328 5111 generic.go:356] "Generic (PLEG): container finished" podID="8bf7d094-ba6e-4c14-b731-c13068585df1" containerID="d0a9bbe12d9d63b469c03867a5799429153313f12e9ab36c577f56a29f4724fc" exitCode=0 Nov 03 20:24:25.396477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.396408 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" Nov 03 20:24:25.396477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.396450 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" event={"ID":"8bf7d094-ba6e-4c14-b731-c13068585df1","Type":"ContainerDied","Data":"d0a9bbe12d9d63b469c03867a5799429153313f12e9ab36c577f56a29f4724fc"} Nov 03 20:24:25.396674 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.396497 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w" event={"ID":"8bf7d094-ba6e-4c14-b731-c13068585df1","Type":"ContainerDied","Data":"af3664d2891452f2be9951b2ae49a48d2256b3a55934f2aad36ac64dbca96c8c"} Nov 03 20:24:25.419329 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.419302 5111 scope.go:122] "RemoveContainer" containerID="6deebfa3bdcf1565192a7b61b9474c838a90a90004bb0239c66e59bebb122765" Nov 03 20:24:25.419815 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:24:25.419788 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6deebfa3bdcf1565192a7b61b9474c838a90a90004bb0239c66e59bebb122765\": container with ID starting with 6deebfa3bdcf1565192a7b61b9474c838a90a90004bb0239c66e59bebb122765 not found: ID does not exist" containerID="6deebfa3bdcf1565192a7b61b9474c838a90a90004bb0239c66e59bebb122765" Nov 03 20:24:25.419944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.419824 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6deebfa3bdcf1565192a7b61b9474c838a90a90004bb0239c66e59bebb122765"} err="failed to get container status \"6deebfa3bdcf1565192a7b61b9474c838a90a90004bb0239c66e59bebb122765\": rpc error: code = NotFound desc = could not find container \"6deebfa3bdcf1565192a7b61b9474c838a90a90004bb0239c66e59bebb122765\": container with ID starting with 6deebfa3bdcf1565192a7b61b9474c838a90a90004bb0239c66e59bebb122765 not found: ID does not exist" Nov 03 20:24:25.419944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.419859 5111 scope.go:122] "RemoveContainer" containerID="d0a9bbe12d9d63b469c03867a5799429153313f12e9ab36c577f56a29f4724fc" Nov 03 20:24:25.434664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.434620 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2"] Nov 03 20:24:25.434789 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.434688 5111 scope.go:122] "RemoveContainer" containerID="d0a9bbe12d9d63b469c03867a5799429153313f12e9ab36c577f56a29f4724fc" Nov 03 20:24:25.435039 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:24:25.435008 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0a9bbe12d9d63b469c03867a5799429153313f12e9ab36c577f56a29f4724fc\": container with ID starting with d0a9bbe12d9d63b469c03867a5799429153313f12e9ab36c577f56a29f4724fc not found: ID does not exist" containerID="d0a9bbe12d9d63b469c03867a5799429153313f12e9ab36c577f56a29f4724fc" Nov 03 20:24:25.435176 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.435047 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0a9bbe12d9d63b469c03867a5799429153313f12e9ab36c577f56a29f4724fc"} err="failed to get container status \"d0a9bbe12d9d63b469c03867a5799429153313f12e9ab36c577f56a29f4724fc\": rpc error: code = NotFound desc = could not find container \"d0a9bbe12d9d63b469c03867a5799429153313f12e9ab36c577f56a29f4724fc\": container with ID starting with d0a9bbe12d9d63b469c03867a5799429153313f12e9ab36c577f56a29f4724fc not found: ID does not exist" Nov 03 20:24:25.439215 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.439194 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-t7bf2"] Nov 03 20:24:25.448703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.448672 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w"] Nov 03 20:24:25.452896 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:25.452868 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-p8c4w"] Nov 03 20:24:26.186081 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:26.186044 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf7d094-ba6e-4c14-b731-c13068585df1" path="/var/lib/kubelet/pods/8bf7d094-ba6e-4c14-b731-c13068585df1/volumes" Nov 03 20:24:26.186822 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:26.186803 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f87478-ca01-41de-b602-f5a831e07913" path="/var/lib/kubelet/pods/c1f87478-ca01-41de-b602-f5a831e07913/volumes" Nov 03 20:24:30.888302 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:30.888264 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:24:30.890687 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:30.890664 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:24:30.893911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:30.893893 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:24:30.896323 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:30.896305 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:24:35.061128 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:24:35.061085 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:24:41.025146 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:41.025110 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn"] Nov 03 20:24:41.032866 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:41.032827 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4"] Nov 03 20:24:41.039136 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:41.039110 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47274747-9jxqn"] Nov 03 20:24:41.044242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:41.044223 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76598cd6-mt9j4"] Nov 03 20:24:42.185137 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:42.185104 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21aa1610-bd37-4be6-8ef5-4217c931f3a5" path="/var/lib/kubelet/pods/21aa1610-bd37-4be6-8ef5-4217c931f3a5/volumes" Nov 03 20:24:42.185739 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:42.185722 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="386d1427-eb8e-4eba-b1ba-f0bac209dfa9" path="/var/lib/kubelet/pods/386d1427-eb8e-4eba-b1ba-f0bac209dfa9/volumes" Nov 03 20:24:53.575895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.575848 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8"] Nov 03 20:24:53.576654 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576638 5111 cpu_manager.go:414] "containerMap: removing container" podUID="c1f87478-ca01-41de-b602-f5a831e07913" containerName="istio-proxy" Nov 03 20:24:53.576712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576654 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f87478-ca01-41de-b602-f5a831e07913" containerName="istio-proxy" Nov 03 20:24:53.576712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576676 5111 cpu_manager.go:414] "containerMap: removing container" podUID="8bf7d094-ba6e-4c14-b731-c13068585df1" containerName="istio-proxy" Nov 03 20:24:53.576712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576683 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf7d094-ba6e-4c14-b731-c13068585df1" containerName="istio-proxy" Nov 03 20:24:53.576712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576697 5111 cpu_manager.go:414] "containerMap: removing container" podUID="fbe4e11b-d14b-4b23-83d4-6815212acbed" containerName="revoke-keys" Nov 03 20:24:53.576712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576702 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe4e11b-d14b-4b23-83d4-6815212acbed" containerName="revoke-keys" Nov 03 20:24:53.577276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576727 5111 cpu_manager.go:414] "containerMap: removing container" podUID="105f9648-6b47-477c-afb1-a41d07095e1b" containerName="payload-pre-processing" Nov 03 20:24:53.577276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576733 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105f9648-6b47-477c-afb1-a41d07095e1b" containerName="payload-pre-processing" Nov 03 20:24:53.577276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576751 5111 cpu_manager.go:414] "containerMap: removing container" podUID="97dad6bc-5f5e-49cf-9fde-1470e77512f0" containerName="payload-processing" Nov 03 20:24:53.577276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576756 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97dad6bc-5f5e-49cf-9fde-1470e77512f0" containerName="payload-processing" Nov 03 20:24:53.577276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576762 5111 cpu_manager.go:414] "containerMap: removing container" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" Nov 03 20:24:53.577276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576766 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" Nov 03 20:24:53.577276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576775 5111 cpu_manager.go:414] "containerMap: removing container" podUID="1ebbac93-2f53-40cb-904f-d8a25729e213" containerName="revoke-keys" Nov 03 20:24:53.577276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576780 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebbac93-2f53-40cb-904f-d8a25729e213" containerName="revoke-keys" Nov 03 20:24:53.577276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576882 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="105f9648-6b47-477c-afb1-a41d07095e1b" containerName="payload-pre-processing" Nov 03 20:24:53.577276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576891 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="97dad6bc-5f5e-49cf-9fde-1470e77512f0" containerName="payload-processing" Nov 03 20:24:53.577276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576900 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="5333c660-c55d-40f0-b0cc-8decef5eeb22" containerName="main" Nov 03 20:24:53.577276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576905 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ebbac93-2f53-40cb-904f-d8a25729e213" containerName="revoke-keys" Nov 03 20:24:53.577276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576912 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f87478-ca01-41de-b602-f5a831e07913" containerName="istio-proxy" Nov 03 20:24:53.577276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576918 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf7d094-ba6e-4c14-b731-c13068585df1" containerName="istio-proxy" Nov 03 20:24:53.577276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.576925 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe4e11b-d14b-4b23-83d4-6815212acbed" containerName="revoke-keys" Nov 03 20:24:53.582353 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.582236 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" Nov 03 20:24:53.585777 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.585756 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:24:53.586954 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.586798 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:24:53.588803 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.588775 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8"] Nov 03 20:24:53.717715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.717675 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fldzx\" (UniqueName: \"kubernetes.io/projected/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-kube-api-access-fldzx\") pod \"maas-api-revoke-sub-11c4b5a4-kjrr8\" (UID: \"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" Nov 03 20:24:53.717957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.717751 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11c4b5a4-kjrr8\" (UID: \"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" Nov 03 20:24:53.717957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.717865 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11c4b5a4-kjrr8\" (UID: \"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" Nov 03 20:24:53.820092 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.820051 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fldzx\" (UniqueName: \"kubernetes.io/projected/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-kube-api-access-fldzx\") pod \"maas-api-revoke-sub-11c4b5a4-kjrr8\" (UID: \"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" Nov 03 20:24:53.820294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.820130 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11c4b5a4-kjrr8\" (UID: \"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" Nov 03 20:24:53.820294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.820187 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11c4b5a4-kjrr8\" (UID: \"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" Nov 03 20:24:53.820867 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.820839 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11c4b5a4-kjrr8\" (UID: \"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" Nov 03 20:24:53.822694 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.822671 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11c4b5a4-kjrr8\" (UID: \"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" Nov 03 20:24:53.827781 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.827725 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fldzx\" (UniqueName: \"kubernetes.io/projected/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-kube-api-access-fldzx\") pod \"maas-api-revoke-sub-11c4b5a4-kjrr8\" (UID: \"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" Nov 03 20:24:53.905149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:53.905106 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" Nov 03 20:24:54.196760 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:24:54.195983 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff3ad5e_85a6_4f86_944c_bdeb4197d1c2.slice/crio-5ce5ce0fda6a694f308010a52b8186e7f1a2bf33a6ccdce3fc542aa98ba5452f WatchSource:0}: Error finding container 5ce5ce0fda6a694f308010a52b8186e7f1a2bf33a6ccdce3fc542aa98ba5452f: Status 404 returned error can't find the container with id 5ce5ce0fda6a694f308010a52b8186e7f1a2bf33a6ccdce3fc542aa98ba5452f Nov 03 20:24:54.196760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:54.196427 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8"] Nov 03 20:24:54.785571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:54.778053 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" event={"ID":"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2","Type":"ContainerStarted","Data":"31b53ee687201111c2542f7e1bf235950479c4867c4cfa5119935c36117e115e"} Nov 03 20:24:54.785571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:54.778127 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" event={"ID":"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2","Type":"ContainerStarted","Data":"5ce5ce0fda6a694f308010a52b8186e7f1a2bf33a6ccdce3fc542aa98ba5452f"} Nov 03 20:24:54.818841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:54.817102 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" podStartSLOduration=1.817083829 podStartE2EDuration="1.817083829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:54.814106741 +0000 UTC m=+2724.776485178" watchObservedRunningTime="2026-11-03 20:24:54.817083829 +0000 UTC m=+2724.779462267" Nov 03 20:24:55.789163 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:55.789125 5111 generic.go:356] "Generic (PLEG): container finished" podID="6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2" containerID="31b53ee687201111c2542f7e1bf235950479c4867c4cfa5119935c36117e115e" exitCode=0 Nov 03 20:24:55.789648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:55.789215 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" event={"ID":"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2","Type":"ContainerDied","Data":"31b53ee687201111c2542f7e1bf235950479c4867c4cfa5119935c36117e115e"} Nov 03 20:24:56.996735 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:56.996702 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" Nov 03 20:24:57.059718 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:57.059627 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-kube-api-access-fldzx\" (UniqueName: \"kubernetes.io/projected/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-kube-api-access-fldzx\") pod \"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2\" (UID: \"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2\") " Nov 03 20:24:57.059718 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:57.059675 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-maas-api-xks-ca\") pod \"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2\" (UID: \"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2\") " Nov 03 20:24:57.059977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:57.059968 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-maas-api-openshift-ca\") pod \"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2\" (UID: \"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2\") " Nov 03 20:24:57.060654 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:57.060621 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-maas-api-openshift-ca" pod "6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2" (UID: "6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:57.062945 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:57.062910 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-maas-api-xks-ca" pod "6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2" (UID: "6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:24:57.063241 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:57.063212 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-kube-api-access-fldzx" pod "6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2" (UID: "6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2"). InnerVolumeSpecName "kube-api-access-fldzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:57.162451 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:57.162408 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fldzx\" (UniqueName: \"kubernetes.io/projected/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-kube-api-access-fldzx\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:57.162451 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:57.162446 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:57.162736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:57.162460 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:57.819511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:57.819465 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" event={"ID":"6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2","Type":"ContainerDied","Data":"5ce5ce0fda6a694f308010a52b8186e7f1a2bf33a6ccdce3fc542aa98ba5452f"} Nov 03 20:24:57.819511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:57.819499 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8" Nov 03 20:24:57.819757 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:24:57.819544 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ce5ce0fda6a694f308010a52b8186e7f1a2bf33a6ccdce3fc542aa98ba5452f" Nov 03 20:25:04.598506 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.598463 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866"] Nov 03 20:25:04.599346 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.599323 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2" containerName="revoke-keys" Nov 03 20:25:04.599346 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.599345 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2" containerName="revoke-keys" Nov 03 20:25:04.599528 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.599479 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2" containerName="revoke-keys" Nov 03 20:25:04.603337 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.603314 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" Nov 03 20:25:04.620557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.612819 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866"] Nov 03 20:25:04.620557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.613151 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:25:04.620557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.614342 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:25:04.651887 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.651838 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqqcn\" (UniqueName: \"kubernetes.io/projected/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-kube-api-access-bqqcn\") pod \"maas-api-revoke-sub-e1f511b2-2r866\" (UID: \"da3f92a8-fbf6-48e0-bf03-6d4e907e1421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" Nov 03 20:25:04.652068 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.652009 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1f511b2-2r866\" (UID: \"da3f92a8-fbf6-48e0-bf03-6d4e907e1421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" Nov 03 20:25:04.652185 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.652162 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1f511b2-2r866\" (UID: \"da3f92a8-fbf6-48e0-bf03-6d4e907e1421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" Nov 03 20:25:04.774704 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.774658 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqqcn\" (UniqueName: \"kubernetes.io/projected/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-kube-api-access-bqqcn\") pod \"maas-api-revoke-sub-e1f511b2-2r866\" (UID: \"da3f92a8-fbf6-48e0-bf03-6d4e907e1421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" Nov 03 20:25:04.775147 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.775124 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1f511b2-2r866\" (UID: \"da3f92a8-fbf6-48e0-bf03-6d4e907e1421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" Nov 03 20:25:04.775404 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.775385 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1f511b2-2r866\" (UID: \"da3f92a8-fbf6-48e0-bf03-6d4e907e1421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" Nov 03 20:25:04.781645 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.781616 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1f511b2-2r866\" (UID: \"da3f92a8-fbf6-48e0-bf03-6d4e907e1421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" Nov 03 20:25:04.803558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.798290 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqqcn\" (UniqueName: \"kubernetes.io/projected/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-kube-api-access-bqqcn\") pod \"maas-api-revoke-sub-e1f511b2-2r866\" (UID: \"da3f92a8-fbf6-48e0-bf03-6d4e907e1421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" Nov 03 20:25:04.846049 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.845997 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1f511b2-2r866\" (UID: \"da3f92a8-fbf6-48e0-bf03-6d4e907e1421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" Nov 03 20:25:04.931496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:04.931461 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" Nov 03 20:25:05.178249 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:25:05.178160 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:25:05.801412 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:05.801360 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866"] Nov 03 20:25:05.842916 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:25:05.842854 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3f92a8_fbf6_48e0_bf03_6d4e907e1421.slice/crio-353cd9c945e858a1d067f36a0b8db181663a10546dbc88e86059cbd6dea9a2f8 WatchSource:0}: Error finding container 353cd9c945e858a1d067f36a0b8db181663a10546dbc88e86059cbd6dea9a2f8: Status 404 returned error can't find the container with id 353cd9c945e858a1d067f36a0b8db181663a10546dbc88e86059cbd6dea9a2f8 Nov 03 20:25:05.950729 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:05.950684 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" event={"ID":"da3f92a8-fbf6-48e0-bf03-6d4e907e1421","Type":"ContainerStarted","Data":"353cd9c945e858a1d067f36a0b8db181663a10546dbc88e86059cbd6dea9a2f8"} Nov 03 20:25:06.973488 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:06.973431 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" event={"ID":"da3f92a8-fbf6-48e0-bf03-6d4e907e1421","Type":"ContainerStarted","Data":"d23806f37e27ba53a3951c2ca07b09aa2d614ca4452fa1e651ad5419c4ec58d5"} Nov 03 20:25:06.993911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:06.993846 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" podStartSLOduration=2.993825655 podStartE2EDuration="2.993825655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:06.990194045 +0000 UTC m=+2736.952572484" watchObservedRunningTime="2026-11-03 20:25:06.993825655 +0000 UTC m=+2736.956204095" Nov 03 20:25:07.071027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.070989 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c"] Nov 03 20:25:07.075839 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.075810 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c" Nov 03 20:25:07.086251 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.086219 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c"] Nov 03 20:25:07.203022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.202930 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f563427-cgc4c\" (UID: \"d1fc2582-7554-4e8d-ae03-a6864f3a15c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c" Nov 03 20:25:07.203291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.203266 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f563427-cgc4c\" (UID: \"d1fc2582-7554-4e8d-ae03-a6864f3a15c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c" Nov 03 20:25:07.203383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.203314 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjqrc\" (UniqueName: \"kubernetes.io/projected/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-kube-api-access-tjqrc\") pod \"maas-api-revoke-sub-5f563427-cgc4c\" (UID: \"d1fc2582-7554-4e8d-ae03-a6864f3a15c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c" Nov 03 20:25:07.304861 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.304818 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f563427-cgc4c\" (UID: \"d1fc2582-7554-4e8d-ae03-a6864f3a15c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c" Nov 03 20:25:07.304861 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.304863 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjqrc\" (UniqueName: \"kubernetes.io/projected/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-kube-api-access-tjqrc\") pod \"maas-api-revoke-sub-5f563427-cgc4c\" (UID: \"d1fc2582-7554-4e8d-ae03-a6864f3a15c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c" Nov 03 20:25:07.305122 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.304979 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f563427-cgc4c\" (UID: \"d1fc2582-7554-4e8d-ae03-a6864f3a15c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c" Nov 03 20:25:07.305661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.305635 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f563427-cgc4c\" (UID: \"d1fc2582-7554-4e8d-ae03-a6864f3a15c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c" Nov 03 20:25:07.307500 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.307476 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f563427-cgc4c\" (UID: \"d1fc2582-7554-4e8d-ae03-a6864f3a15c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c" Nov 03 20:25:07.313066 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.313041 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjqrc\" (UniqueName: \"kubernetes.io/projected/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-kube-api-access-tjqrc\") pod \"maas-api-revoke-sub-5f563427-cgc4c\" (UID: \"d1fc2582-7554-4e8d-ae03-a6864f3a15c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c" Nov 03 20:25:07.407950 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.407912 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c" Nov 03 20:25:07.727162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.726890 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c"] Nov 03 20:25:07.995636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.995560 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c" event={"ID":"d1fc2582-7554-4e8d-ae03-a6864f3a15c9","Type":"ContainerStarted","Data":"e45a48d9fdc42e88dbbf9ae95719d2c8262735acc13efd16c289c0cf1e6f36dc"} Nov 03 20:25:07.997360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.997332 5111 generic.go:356] "Generic (PLEG): container finished" podID="da3f92a8-fbf6-48e0-bf03-6d4e907e1421" containerID="d23806f37e27ba53a3951c2ca07b09aa2d614ca4452fa1e651ad5419c4ec58d5" exitCode=0 Nov 03 20:25:07.997555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:07.997502 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" event={"ID":"da3f92a8-fbf6-48e0-bf03-6d4e907e1421","Type":"ContainerDied","Data":"d23806f37e27ba53a3951c2ca07b09aa2d614ca4452fa1e651ad5419c4ec58d5"} Nov 03 20:25:09.009088 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.009050 5111 generic.go:356] "Generic (PLEG): container finished" podID="d1fc2582-7554-4e8d-ae03-a6864f3a15c9" containerID="d56c36f7efa98a54048647c8a04adf84e32ed877f79901dfe07ab3656883a1e1" exitCode=0 Nov 03 20:25:09.009630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.009125 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c" event={"ID":"d1fc2582-7554-4e8d-ae03-a6864f3a15c9","Type":"ContainerDied","Data":"d56c36f7efa98a54048647c8a04adf84e32ed877f79901dfe07ab3656883a1e1"} Nov 03 20:25:09.203090 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.201207 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59"] Nov 03 20:25:09.233802 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.233770 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59"] Nov 03 20:25:09.233963 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.233904 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" Nov 03 20:25:09.292811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.292763 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0954b165-7182-489a-b508-c21b4b21b0a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dac3af0e-c5k59\" (UID: \"0954b165-7182-489a-b508-c21b4b21b0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" Nov 03 20:25:09.301749 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.301721 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" Nov 03 20:25:09.395791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.395752 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whk6r\" (UniqueName: \"kubernetes.io/projected/0954b165-7182-489a-b508-c21b4b21b0a9-kube-api-access-whk6r\") pod \"maas-api-revoke-sub-dac3af0e-c5k59\" (UID: \"0954b165-7182-489a-b508-c21b4b21b0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" Nov 03 20:25:09.395984 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.395861 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0954b165-7182-489a-b508-c21b4b21b0a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dac3af0e-c5k59\" (UID: \"0954b165-7182-489a-b508-c21b4b21b0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" Nov 03 20:25:09.395984 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.395944 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0954b165-7182-489a-b508-c21b4b21b0a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dac3af0e-c5k59\" (UID: \"0954b165-7182-489a-b508-c21b4b21b0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" Nov 03 20:25:09.396648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.396624 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0954b165-7182-489a-b508-c21b4b21b0a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dac3af0e-c5k59\" (UID: \"0954b165-7182-489a-b508-c21b4b21b0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" Nov 03 20:25:09.497444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.497350 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-kube-api-access-bqqcn\" (UniqueName: \"kubernetes.io/projected/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-kube-api-access-bqqcn\") pod \"da3f92a8-fbf6-48e0-bf03-6d4e907e1421\" (UID: \"da3f92a8-fbf6-48e0-bf03-6d4e907e1421\") " Nov 03 20:25:09.497679 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.497583 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-maas-api-xks-ca\") pod \"da3f92a8-fbf6-48e0-bf03-6d4e907e1421\" (UID: \"da3f92a8-fbf6-48e0-bf03-6d4e907e1421\") " Nov 03 20:25:09.497679 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.497656 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-maas-api-openshift-ca\") pod \"da3f92a8-fbf6-48e0-bf03-6d4e907e1421\" (UID: \"da3f92a8-fbf6-48e0-bf03-6d4e907e1421\") " Nov 03 20:25:09.498001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.497977 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-maas-api-openshift-ca" pod "da3f92a8-fbf6-48e0-bf03-6d4e907e1421" (UID: "da3f92a8-fbf6-48e0-bf03-6d4e907e1421"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:09.498202 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.498179 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whk6r\" (UniqueName: \"kubernetes.io/projected/0954b165-7182-489a-b508-c21b4b21b0a9-kube-api-access-whk6r\") pod \"maas-api-revoke-sub-dac3af0e-c5k59\" (UID: \"0954b165-7182-489a-b508-c21b4b21b0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" Nov 03 20:25:09.498292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.498278 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0954b165-7182-489a-b508-c21b4b21b0a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dac3af0e-c5k59\" (UID: \"0954b165-7182-489a-b508-c21b4b21b0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" Nov 03 20:25:09.498487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.498467 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:09.499961 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.499932 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-kube-api-access-bqqcn" pod "da3f92a8-fbf6-48e0-bf03-6d4e907e1421" (UID: "da3f92a8-fbf6-48e0-bf03-6d4e907e1421"). InnerVolumeSpecName "kube-api-access-bqqcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:09.499961 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.499948 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-maas-api-xks-ca" pod "da3f92a8-fbf6-48e0-bf03-6d4e907e1421" (UID: "da3f92a8-fbf6-48e0-bf03-6d4e907e1421"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:09.500812 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.500789 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0954b165-7182-489a-b508-c21b4b21b0a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dac3af0e-c5k59\" (UID: \"0954b165-7182-489a-b508-c21b4b21b0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" Nov 03 20:25:09.506156 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.506132 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whk6r\" (UniqueName: \"kubernetes.io/projected/0954b165-7182-489a-b508-c21b4b21b0a9-kube-api-access-whk6r\") pod \"maas-api-revoke-sub-dac3af0e-c5k59\" (UID: \"0954b165-7182-489a-b508-c21b4b21b0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" Nov 03 20:25:09.597050 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.597019 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" Nov 03 20:25:09.600110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.600081 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:09.600110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.600111 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqqcn\" (UniqueName: \"kubernetes.io/projected/da3f92a8-fbf6-48e0-bf03-6d4e907e1421-kube-api-access-bqqcn\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:09.886404 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:09.886370 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59"] Nov 03 20:25:09.886566 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:25:09.886446 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0954b165_7182_489a_b508_c21b4b21b0a9.slice/crio-cf0cc3731a0a6a60d78dc73445ae8838163ade9c2d0cb304cefc17b63bcbcd56 WatchSource:0}: Error finding container cf0cc3731a0a6a60d78dc73445ae8838163ade9c2d0cb304cefc17b63bcbcd56: Status 404 returned error can't find the container with id cf0cc3731a0a6a60d78dc73445ae8838163ade9c2d0cb304cefc17b63bcbcd56 Nov 03 20:25:10.018454 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:10.018418 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" event={"ID":"0954b165-7182-489a-b508-c21b4b21b0a9","Type":"ContainerStarted","Data":"cf0cc3731a0a6a60d78dc73445ae8838163ade9c2d0cb304cefc17b63bcbcd56"} Nov 03 20:25:10.020191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:10.020161 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" Nov 03 20:25:10.020323 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:10.020191 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866" event={"ID":"da3f92a8-fbf6-48e0-bf03-6d4e907e1421","Type":"ContainerDied","Data":"353cd9c945e858a1d067f36a0b8db181663a10546dbc88e86059cbd6dea9a2f8"} Nov 03 20:25:10.020323 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:10.020244 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="353cd9c945e858a1d067f36a0b8db181663a10546dbc88e86059cbd6dea9a2f8" Nov 03 20:25:10.191400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:10.191374 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c" Nov 03 20:25:10.259030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:10.258956 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-kube-api-access-tjqrc\" (UniqueName: \"kubernetes.io/projected/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-kube-api-access-tjqrc\") pod \"d1fc2582-7554-4e8d-ae03-a6864f3a15c9\" (UID: \"d1fc2582-7554-4e8d-ae03-a6864f3a15c9\") " Nov 03 20:25:10.259169 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:10.259065 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-maas-api-xks-ca\") pod \"d1fc2582-7554-4e8d-ae03-a6864f3a15c9\" (UID: \"d1fc2582-7554-4e8d-ae03-a6864f3a15c9\") " Nov 03 20:25:10.259260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:10.259234 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-maas-api-openshift-ca\") pod \"d1fc2582-7554-4e8d-ae03-a6864f3a15c9\" (UID: \"d1fc2582-7554-4e8d-ae03-a6864f3a15c9\") " Nov 03 20:25:10.259773 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:10.259748 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-maas-api-openshift-ca" pod "d1fc2582-7554-4e8d-ae03-a6864f3a15c9" (UID: "d1fc2582-7554-4e8d-ae03-a6864f3a15c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:10.260590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:10.260454 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:10.262602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:10.262524 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-maas-api-xks-ca" pod "d1fc2582-7554-4e8d-ae03-a6864f3a15c9" (UID: "d1fc2582-7554-4e8d-ae03-a6864f3a15c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:10.263067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:10.263042 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-kube-api-access-tjqrc" pod "d1fc2582-7554-4e8d-ae03-a6864f3a15c9" (UID: "d1fc2582-7554-4e8d-ae03-a6864f3a15c9"). InnerVolumeSpecName "kube-api-access-tjqrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:10.362076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:10.362044 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjqrc\" (UniqueName: \"kubernetes.io/projected/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-kube-api-access-tjqrc\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:10.362213 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:10.362081 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1fc2582-7554-4e8d-ae03-a6864f3a15c9-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:11.030813 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:11.030773 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" event={"ID":"0954b165-7182-489a-b508-c21b4b21b0a9","Type":"ContainerStarted","Data":"ed912031179f70e1612c06904f09c00f1466848173471470aa2b3c12d1651d5d"} Nov 03 20:25:11.032785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:11.032758 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c" Nov 03 20:25:11.032941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:11.032793 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c" event={"ID":"d1fc2582-7554-4e8d-ae03-a6864f3a15c9","Type":"ContainerDied","Data":"e45a48d9fdc42e88dbbf9ae95719d2c8262735acc13efd16c289c0cf1e6f36dc"} Nov 03 20:25:11.032941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:11.032840 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e45a48d9fdc42e88dbbf9ae95719d2c8262735acc13efd16c289c0cf1e6f36dc" Nov 03 20:25:11.047913 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:11.047859 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" podStartSLOduration=2.047830228 podStartE2EDuration="2.047830228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:11.046429025 +0000 UTC m=+2741.008807463" watchObservedRunningTime="2026-11-03 20:25:11.047830228 +0000 UTC m=+2741.010208666" Nov 03 20:25:12.044833 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:12.044801 5111 generic.go:356] "Generic (PLEG): container finished" podID="0954b165-7182-489a-b508-c21b4b21b0a9" containerID="ed912031179f70e1612c06904f09c00f1466848173471470aa2b3c12d1651d5d" exitCode=0 Nov 03 20:25:12.045292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:12.044884 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" event={"ID":"0954b165-7182-489a-b508-c21b4b21b0a9","Type":"ContainerDied","Data":"ed912031179f70e1612c06904f09c00f1466848173471470aa2b3c12d1651d5d"} Nov 03 20:25:13.253076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:13.253041 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" Nov 03 20:25:13.422314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:13.422279 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0954b165-7182-489a-b508-c21b4b21b0a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0954b165-7182-489a-b508-c21b4b21b0a9-maas-api-openshift-ca\") pod \"0954b165-7182-489a-b508-c21b4b21b0a9\" (UID: \"0954b165-7182-489a-b508-c21b4b21b0a9\") " Nov 03 20:25:13.422509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:13.422399 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0954b165-7182-489a-b508-c21b4b21b0a9-kube-api-access-whk6r\" (UniqueName: \"kubernetes.io/projected/0954b165-7182-489a-b508-c21b4b21b0a9-kube-api-access-whk6r\") pod \"0954b165-7182-489a-b508-c21b4b21b0a9\" (UID: \"0954b165-7182-489a-b508-c21b4b21b0a9\") " Nov 03 20:25:13.422509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:13.422450 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0954b165-7182-489a-b508-c21b4b21b0a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0954b165-7182-489a-b508-c21b4b21b0a9-maas-api-xks-ca\") pod \"0954b165-7182-489a-b508-c21b4b21b0a9\" (UID: \"0954b165-7182-489a-b508-c21b4b21b0a9\") " Nov 03 20:25:13.422787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:13.422761 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0954b165-7182-489a-b508-c21b4b21b0a9-maas-api-openshift-ca" pod "0954b165-7182-489a-b508-c21b4b21b0a9" (UID: "0954b165-7182-489a-b508-c21b4b21b0a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:13.423279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:13.423255 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0954b165-7182-489a-b508-c21b4b21b0a9-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:13.424664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:13.424635 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0954b165-7182-489a-b508-c21b4b21b0a9-maas-api-xks-ca" pod "0954b165-7182-489a-b508-c21b4b21b0a9" (UID: "0954b165-7182-489a-b508-c21b4b21b0a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:13.424767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:13.424680 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0954b165-7182-489a-b508-c21b4b21b0a9-kube-api-access-whk6r" pod "0954b165-7182-489a-b508-c21b4b21b0a9" (UID: "0954b165-7182-489a-b508-c21b4b21b0a9"). InnerVolumeSpecName "kube-api-access-whk6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:13.525690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:13.525650 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whk6r\" (UniqueName: \"kubernetes.io/projected/0954b165-7182-489a-b508-c21b4b21b0a9-kube-api-access-whk6r\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:13.525690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:13.525690 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0954b165-7182-489a-b508-c21b4b21b0a9-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:14.065633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:14.065596 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" event={"ID":"0954b165-7182-489a-b508-c21b4b21b0a9","Type":"ContainerDied","Data":"cf0cc3731a0a6a60d78dc73445ae8838163ade9c2d0cb304cefc17b63bcbcd56"} Nov 03 20:25:14.065791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:14.065644 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59" Nov 03 20:25:14.065791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:14.065653 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf0cc3731a0a6a60d78dc73445ae8838163ade9c2d0cb304cefc17b63bcbcd56" Nov 03 20:25:15.428555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.428272 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv"] Nov 03 20:25:15.432498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.429443 5111 cpu_manager.go:414] "containerMap: removing container" podUID="d1fc2582-7554-4e8d-ae03-a6864f3a15c9" containerName="revoke-keys" Nov 03 20:25:15.432498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.429470 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1fc2582-7554-4e8d-ae03-a6864f3a15c9" containerName="revoke-keys" Nov 03 20:25:15.432498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.429556 5111 cpu_manager.go:414] "containerMap: removing container" podUID="da3f92a8-fbf6-48e0-bf03-6d4e907e1421" containerName="revoke-keys" Nov 03 20:25:15.432498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.429565 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3f92a8-fbf6-48e0-bf03-6d4e907e1421" containerName="revoke-keys" Nov 03 20:25:15.432498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.429584 5111 cpu_manager.go:414] "containerMap: removing container" podUID="0954b165-7182-489a-b508-c21b4b21b0a9" containerName="revoke-keys" Nov 03 20:25:15.432498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.429592 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0954b165-7182-489a-b508-c21b4b21b0a9" containerName="revoke-keys" Nov 03 20:25:15.432498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.429769 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3f92a8-fbf6-48e0-bf03-6d4e907e1421" containerName="revoke-keys" Nov 03 20:25:15.432498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.429783 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="0954b165-7182-489a-b508-c21b4b21b0a9" containerName="revoke-keys" Nov 03 20:25:15.432498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.429799 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1fc2582-7554-4e8d-ae03-a6864f3a15c9" containerName="revoke-keys" Nov 03 20:25:15.433839 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.433811 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" Nov 03 20:25:15.447160 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.446900 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:25:15.447160 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.446999 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:25:15.454549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.450736 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv"] Nov 03 20:25:15.494683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.494607 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mdn2\" (UniqueName: \"kubernetes.io/projected/df9aa3f5-199d-4126-a993-39240f7a098c-kube-api-access-6mdn2\") pod \"maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv\" (UID: \"df9aa3f5-199d-4126-a993-39240f7a098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" Nov 03 20:25:15.494683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.494681 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9aa3f5-199d-4126-a993-39240f7a098c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv\" (UID: \"df9aa3f5-199d-4126-a993-39240f7a098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" Nov 03 20:25:15.494969 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.494782 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9aa3f5-199d-4126-a993-39240f7a098c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv\" (UID: \"df9aa3f5-199d-4126-a993-39240f7a098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" Nov 03 20:25:15.596367 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.596324 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mdn2\" (UniqueName: \"kubernetes.io/projected/df9aa3f5-199d-4126-a993-39240f7a098c-kube-api-access-6mdn2\") pod \"maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv\" (UID: \"df9aa3f5-199d-4126-a993-39240f7a098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" Nov 03 20:25:15.596621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.596394 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9aa3f5-199d-4126-a993-39240f7a098c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv\" (UID: \"df9aa3f5-199d-4126-a993-39240f7a098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" Nov 03 20:25:15.596621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.596430 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9aa3f5-199d-4126-a993-39240f7a098c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv\" (UID: \"df9aa3f5-199d-4126-a993-39240f7a098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" Nov 03 20:25:15.597182 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.597162 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9aa3f5-199d-4126-a993-39240f7a098c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv\" (UID: \"df9aa3f5-199d-4126-a993-39240f7a098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" Nov 03 20:25:15.598966 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.598947 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9aa3f5-199d-4126-a993-39240f7a098c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv\" (UID: \"df9aa3f5-199d-4126-a993-39240f7a098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" Nov 03 20:25:15.603970 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.603945 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mdn2\" (UniqueName: \"kubernetes.io/projected/df9aa3f5-199d-4126-a993-39240f7a098c-kube-api-access-6mdn2\") pod \"maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv\" (UID: \"df9aa3f5-199d-4126-a993-39240f7a098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" Nov 03 20:25:15.770135 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:15.770053 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" Nov 03 20:25:16.141112 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:16.141079 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv"] Nov 03 20:25:16.141907 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:25:16.141881 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9aa3f5_199d_4126_a993_39240f7a098c.slice/crio-14400c893ad7a3bb308552a8bcbfa129a430b980af72b5246d5b563cf40ea0cd WatchSource:0}: Error finding container 14400c893ad7a3bb308552a8bcbfa129a430b980af72b5246d5b563cf40ea0cd: Status 404 returned error can't find the container with id 14400c893ad7a3bb308552a8bcbfa129a430b980af72b5246d5b563cf40ea0cd Nov 03 20:25:17.093997 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.093954 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" event={"ID":"df9aa3f5-199d-4126-a993-39240f7a098c","Type":"ContainerStarted","Data":"d161f4b844a1bf1424c00e9e941685142a22f0094ebbbc0ddbe9e24fd0b9588d"} Nov 03 20:25:17.094410 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.094054 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" event={"ID":"df9aa3f5-199d-4126-a993-39240f7a098c","Type":"ContainerStarted","Data":"14400c893ad7a3bb308552a8bcbfa129a430b980af72b5246d5b563cf40ea0cd"} Nov 03 20:25:17.111198 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.111133 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" podStartSLOduration=2.111089602 podStartE2EDuration="2.111089602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:17.110355097 +0000 UTC m=+2747.072733537" watchObservedRunningTime="2026-11-03 20:25:17.111089602 +0000 UTC m=+2747.073468041" Nov 03 20:25:17.374931 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.374897 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7"] Nov 03 20:25:17.380211 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.380186 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7" Nov 03 20:25:17.385321 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.385293 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7"] Nov 03 20:25:17.435367 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.435330 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1e9568-9815-46be-bfff-746ed30973e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a625455e-vrdt7\" (UID: \"3d1e9568-9815-46be-bfff-746ed30973e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7" Nov 03 20:25:17.435667 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.435632 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnbrm\" (UniqueName: \"kubernetes.io/projected/3d1e9568-9815-46be-bfff-746ed30973e1-kube-api-access-rnbrm\") pod \"maas-api-revoke-sub-a625455e-vrdt7\" (UID: \"3d1e9568-9815-46be-bfff-746ed30973e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7" Nov 03 20:25:17.435870 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.435824 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1e9568-9815-46be-bfff-746ed30973e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a625455e-vrdt7\" (UID: \"3d1e9568-9815-46be-bfff-746ed30973e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7" Nov 03 20:25:17.537311 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.537276 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnbrm\" (UniqueName: \"kubernetes.io/projected/3d1e9568-9815-46be-bfff-746ed30973e1-kube-api-access-rnbrm\") pod \"maas-api-revoke-sub-a625455e-vrdt7\" (UID: \"3d1e9568-9815-46be-bfff-746ed30973e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7" Nov 03 20:25:17.537504 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.537371 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1e9568-9815-46be-bfff-746ed30973e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a625455e-vrdt7\" (UID: \"3d1e9568-9815-46be-bfff-746ed30973e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7" Nov 03 20:25:17.537648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.537596 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1e9568-9815-46be-bfff-746ed30973e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a625455e-vrdt7\" (UID: \"3d1e9568-9815-46be-bfff-746ed30973e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7" Nov 03 20:25:17.538095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.538074 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1e9568-9815-46be-bfff-746ed30973e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a625455e-vrdt7\" (UID: \"3d1e9568-9815-46be-bfff-746ed30973e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7" Nov 03 20:25:17.540135 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.540116 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1e9568-9815-46be-bfff-746ed30973e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a625455e-vrdt7\" (UID: \"3d1e9568-9815-46be-bfff-746ed30973e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7" Nov 03 20:25:17.546042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.546015 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnbrm\" (UniqueName: \"kubernetes.io/projected/3d1e9568-9815-46be-bfff-746ed30973e1-kube-api-access-rnbrm\") pod \"maas-api-revoke-sub-a625455e-vrdt7\" (UID: \"3d1e9568-9815-46be-bfff-746ed30973e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7" Nov 03 20:25:17.700161 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.700075 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7" Nov 03 20:25:17.981957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:17.981924 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7"] Nov 03 20:25:17.982078 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:25:17.982034 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1e9568_9815_46be_bfff_746ed30973e1.slice/crio-89faeaab9182a587105783f4203c4e6999dd246d27c7d1b78f541f028195a9c8 WatchSource:0}: Error finding container 89faeaab9182a587105783f4203c4e6999dd246d27c7d1b78f541f028195a9c8: Status 404 returned error can't find the container with id 89faeaab9182a587105783f4203c4e6999dd246d27c7d1b78f541f028195a9c8 Nov 03 20:25:18.133439 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:18.131182 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7" event={"ID":"3d1e9568-9815-46be-bfff-746ed30973e1","Type":"ContainerStarted","Data":"89faeaab9182a587105783f4203c4e6999dd246d27c7d1b78f541f028195a9c8"} Nov 03 20:25:18.155666 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:18.136341 5111 generic.go:356] "Generic (PLEG): container finished" podID="df9aa3f5-199d-4126-a993-39240f7a098c" containerID="d161f4b844a1bf1424c00e9e941685142a22f0094ebbbc0ddbe9e24fd0b9588d" exitCode=0 Nov 03 20:25:18.155666 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:18.136681 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" event={"ID":"df9aa3f5-199d-4126-a993-39240f7a098c","Type":"ContainerDied","Data":"d161f4b844a1bf1424c00e9e941685142a22f0094ebbbc0ddbe9e24fd0b9588d"} Nov 03 20:25:19.147509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.147474 5111 generic.go:356] "Generic (PLEG): container finished" podID="3d1e9568-9815-46be-bfff-746ed30973e1" containerID="e41bcf9aa4d1370f42e4c4c9aa49707111174c97d2edef070c5ed1bee30d4d2e" exitCode=0 Nov 03 20:25:19.148010 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.147558 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7" event={"ID":"3d1e9568-9815-46be-bfff-746ed30973e1","Type":"ContainerDied","Data":"e41bcf9aa4d1370f42e4c4c9aa49707111174c97d2edef070c5ed1bee30d4d2e"} Nov 03 20:25:19.328332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.328300 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" Nov 03 20:25:19.369565 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.369517 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9aa3f5-199d-4126-a993-39240f7a098c-kube-api-access-6mdn2\" (UniqueName: \"kubernetes.io/projected/df9aa3f5-199d-4126-a993-39240f7a098c-kube-api-access-6mdn2\") pod \"df9aa3f5-199d-4126-a993-39240f7a098c\" (UID: \"df9aa3f5-199d-4126-a993-39240f7a098c\") " Nov 03 20:25:19.369771 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.369700 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df9aa3f5-199d-4126-a993-39240f7a098c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9aa3f5-199d-4126-a993-39240f7a098c-maas-api-xks-ca\") pod \"df9aa3f5-199d-4126-a993-39240f7a098c\" (UID: \"df9aa3f5-199d-4126-a993-39240f7a098c\") " Nov 03 20:25:19.369861 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.369796 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df9aa3f5-199d-4126-a993-39240f7a098c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9aa3f5-199d-4126-a993-39240f7a098c-maas-api-openshift-ca\") pod \"df9aa3f5-199d-4126-a993-39240f7a098c\" (UID: \"df9aa3f5-199d-4126-a993-39240f7a098c\") " Nov 03 20:25:19.370727 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.370206 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9aa3f5-199d-4126-a993-39240f7a098c-maas-api-openshift-ca" pod "df9aa3f5-199d-4126-a993-39240f7a098c" (UID: "df9aa3f5-199d-4126-a993-39240f7a098c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:19.370727 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.370677 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9aa3f5-199d-4126-a993-39240f7a098c-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:19.372738 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.372703 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9aa3f5-199d-4126-a993-39240f7a098c-maas-api-xks-ca" pod "df9aa3f5-199d-4126-a993-39240f7a098c" (UID: "df9aa3f5-199d-4126-a993-39240f7a098c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:19.372738 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.372716 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9aa3f5-199d-4126-a993-39240f7a098c-kube-api-access-6mdn2" pod "df9aa3f5-199d-4126-a993-39240f7a098c" (UID: "df9aa3f5-199d-4126-a993-39240f7a098c"). InnerVolumeSpecName "kube-api-access-6mdn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:19.426624 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.426582 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8"] Nov 03 20:25:19.427449 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.427434 5111 cpu_manager.go:414] "containerMap: removing container" podUID="df9aa3f5-199d-4126-a993-39240f7a098c" containerName="revoke-keys" Nov 03 20:25:19.427509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.427451 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df9aa3f5-199d-4126-a993-39240f7a098c" containerName="revoke-keys" Nov 03 20:25:19.427653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.427640 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="df9aa3f5-199d-4126-a993-39240f7a098c" containerName="revoke-keys" Nov 03 20:25:19.431635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.431614 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" Nov 03 20:25:19.441381 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.441354 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8"] Nov 03 20:25:19.472035 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.471943 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz5w6\" (UniqueName: \"kubernetes.io/projected/95b9f28c-e165-436f-a5a5-ae89d046297f-kube-api-access-gz5w6\") pod \"maas-api-revoke-sub-2a5d8dc4-ljpm8\" (UID: \"95b9f28c-e165-436f-a5a5-ae89d046297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" Nov 03 20:25:19.472035 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.472022 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b9f28c-e165-436f-a5a5-ae89d046297f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a5d8dc4-ljpm8\" (UID: \"95b9f28c-e165-436f-a5a5-ae89d046297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" Nov 03 20:25:19.472251 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.472173 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b9f28c-e165-436f-a5a5-ae89d046297f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a5d8dc4-ljpm8\" (UID: \"95b9f28c-e165-436f-a5a5-ae89d046297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" Nov 03 20:25:19.472331 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.472299 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mdn2\" (UniqueName: \"kubernetes.io/projected/df9aa3f5-199d-4126-a993-39240f7a098c-kube-api-access-6mdn2\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:19.472331 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.472318 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9aa3f5-199d-4126-a993-39240f7a098c-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:19.574300 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.574260 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b9f28c-e165-436f-a5a5-ae89d046297f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a5d8dc4-ljpm8\" (UID: \"95b9f28c-e165-436f-a5a5-ae89d046297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" Nov 03 20:25:19.574643 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.574599 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5w6\" (UniqueName: \"kubernetes.io/projected/95b9f28c-e165-436f-a5a5-ae89d046297f-kube-api-access-gz5w6\") pod \"maas-api-revoke-sub-2a5d8dc4-ljpm8\" (UID: \"95b9f28c-e165-436f-a5a5-ae89d046297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" Nov 03 20:25:19.574769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.574708 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b9f28c-e165-436f-a5a5-ae89d046297f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a5d8dc4-ljpm8\" (UID: \"95b9f28c-e165-436f-a5a5-ae89d046297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" Nov 03 20:25:19.575493 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.575470 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b9f28c-e165-436f-a5a5-ae89d046297f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a5d8dc4-ljpm8\" (UID: \"95b9f28c-e165-436f-a5a5-ae89d046297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" Nov 03 20:25:19.577268 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.577249 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b9f28c-e165-436f-a5a5-ae89d046297f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a5d8dc4-ljpm8\" (UID: \"95b9f28c-e165-436f-a5a5-ae89d046297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" Nov 03 20:25:19.582175 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.582151 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz5w6\" (UniqueName: \"kubernetes.io/projected/95b9f28c-e165-436f-a5a5-ae89d046297f-kube-api-access-gz5w6\") pod \"maas-api-revoke-sub-2a5d8dc4-ljpm8\" (UID: \"95b9f28c-e165-436f-a5a5-ae89d046297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" Nov 03 20:25:19.753805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:19.753061 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" Nov 03 20:25:20.024146 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.024061 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d"] Nov 03 20:25:20.028762 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.028729 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e920ee1-bmf6d"] Nov 03 20:25:20.048213 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.048162 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8"] Nov 03 20:25:20.050984 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:25:20.050959 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b9f28c_e165_436f_a5a5_ae89d046297f.slice/crio-1ae541a12934e19747bf0e3a77e51bada5a586f56b6e977ecaf29bca022cc42c WatchSource:0}: Error finding container 1ae541a12934e19747bf0e3a77e51bada5a586f56b6e977ecaf29bca022cc42c: Status 404 returned error can't find the container with id 1ae541a12934e19747bf0e3a77e51bada5a586f56b6e977ecaf29bca022cc42c Nov 03 20:25:20.157480 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.157444 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" event={"ID":"95b9f28c-e165-436f-a5a5-ae89d046297f","Type":"ContainerStarted","Data":"1ae541a12934e19747bf0e3a77e51bada5a586f56b6e977ecaf29bca022cc42c"} Nov 03 20:25:20.159242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.159202 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" event={"ID":"df9aa3f5-199d-4126-a993-39240f7a098c","Type":"ContainerDied","Data":"14400c893ad7a3bb308552a8bcbfa129a430b980af72b5246d5b563cf40ea0cd"} Nov 03 20:25:20.159392 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.159270 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14400c893ad7a3bb308552a8bcbfa129a430b980af72b5246d5b563cf40ea0cd" Nov 03 20:25:20.159392 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.159337 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv" Nov 03 20:25:20.203028 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.202995 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85998d26-9a30-423e-91c8-0926e2f724ae" path="/var/lib/kubelet/pods/85998d26-9a30-423e-91c8-0926e2f724ae/volumes" Nov 03 20:25:20.476556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.470431 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7" Nov 03 20:25:20.596428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.596336 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d1e9568-9815-46be-bfff-746ed30973e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1e9568-9815-46be-bfff-746ed30973e1-maas-api-xks-ca\") pod \"3d1e9568-9815-46be-bfff-746ed30973e1\" (UID: \"3d1e9568-9815-46be-bfff-746ed30973e1\") " Nov 03 20:25:20.596658 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.596470 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1e9568-9815-46be-bfff-746ed30973e1-kube-api-access-rnbrm\" (UniqueName: \"kubernetes.io/projected/3d1e9568-9815-46be-bfff-746ed30973e1-kube-api-access-rnbrm\") pod \"3d1e9568-9815-46be-bfff-746ed30973e1\" (UID: \"3d1e9568-9815-46be-bfff-746ed30973e1\") " Nov 03 20:25:20.596752 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.596680 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d1e9568-9815-46be-bfff-746ed30973e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1e9568-9815-46be-bfff-746ed30973e1-maas-api-openshift-ca\") pod \"3d1e9568-9815-46be-bfff-746ed30973e1\" (UID: \"3d1e9568-9815-46be-bfff-746ed30973e1\") " Nov 03 20:25:20.597040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.597004 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1e9568-9815-46be-bfff-746ed30973e1-maas-api-openshift-ca" pod "3d1e9568-9815-46be-bfff-746ed30973e1" (UID: "3d1e9568-9815-46be-bfff-746ed30973e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:20.597406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.597382 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1e9568-9815-46be-bfff-746ed30973e1-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:20.598721 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.598698 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d1e9568-9815-46be-bfff-746ed30973e1-maas-api-xks-ca" pod "3d1e9568-9815-46be-bfff-746ed30973e1" (UID: "3d1e9568-9815-46be-bfff-746ed30973e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:20.598806 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.598735 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1e9568-9815-46be-bfff-746ed30973e1-kube-api-access-rnbrm" pod "3d1e9568-9815-46be-bfff-746ed30973e1" (UID: "3d1e9568-9815-46be-bfff-746ed30973e1"). InnerVolumeSpecName "kube-api-access-rnbrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:20.698955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.698922 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1e9568-9815-46be-bfff-746ed30973e1-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:20.698955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.698953 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnbrm\" (UniqueName: \"kubernetes.io/projected/3d1e9568-9815-46be-bfff-746ed30973e1-kube-api-access-rnbrm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:20.707703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.707672 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn"] Nov 03 20:25:20.708525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.708502 5111 cpu_manager.go:414] "containerMap: removing container" podUID="3d1e9568-9815-46be-bfff-746ed30973e1" containerName="revoke-keys" Nov 03 20:25:20.708525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.708526 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1e9568-9815-46be-bfff-746ed30973e1" containerName="revoke-keys" Nov 03 20:25:20.708817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.708745 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1e9568-9815-46be-bfff-746ed30973e1" containerName="revoke-keys" Nov 03 20:25:20.712704 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.712685 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.716638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.716615 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-27d3ad-openshift-default-dockercfg-wt2s4\"" Nov 03 20:25:20.722724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.722698 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn"] Nov 03 20:25:20.800253 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.800212 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-token\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.800448 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.800261 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh7qs\" (UniqueName: \"kubernetes.io/projected/d5b33145-a82b-4519-ab6a-c7ce0248992f-kube-api-access-jh7qs\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.800448 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.800362 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-data\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.800560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.800467 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-workload-socket\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.800629 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.800600 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b33145-a82b-4519-ab6a-c7ce0248992f-istiod-ca-cert\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.800799 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.800783 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-podinfo\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.800862 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.800816 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-envoy\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.800903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.800879 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-workload-certs\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.800942 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.800910 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-credential-socket\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.902701 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.902666 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-token\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.902865 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.902715 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh7qs\" (UniqueName: \"kubernetes.io/projected/d5b33145-a82b-4519-ab6a-c7ce0248992f-kube-api-access-jh7qs\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.902944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.902866 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-data\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.903007 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.902940 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-workload-socket\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.903099 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.903074 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b33145-a82b-4519-ab6a-c7ce0248992f-istiod-ca-cert\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.903305 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.903262 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-podinfo\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.903462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.903306 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-envoy\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.903462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.903348 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-workload-certs\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.903462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.903358 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-data\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.903462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.903396 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-credential-socket\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.903462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.903394 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-workload-socket\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.903802 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.903693 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-workload-certs\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.903802 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.903768 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-credential-socket\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.904208 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.904183 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b33145-a82b-4519-ab6a-c7ce0248992f-istiod-ca-cert\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.905689 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.905669 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-envoy\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.905868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.905852 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-podinfo\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.910508 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.910483 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-token\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:20.910758 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:20.910741 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh7qs\" (UniqueName: \"kubernetes.io/projected/d5b33145-a82b-4519-ab6a-c7ce0248992f-kube-api-access-jh7qs\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:21.025058 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:21.025019 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj"] Nov 03 20:25:21.029490 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:21.029462 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-427da27c-dkffj"] Nov 03 20:25:21.031589 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:21.031074 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:21.182769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:21.180132 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7" event={"ID":"3d1e9568-9815-46be-bfff-746ed30973e1","Type":"ContainerDied","Data":"89faeaab9182a587105783f4203c4e6999dd246d27c7d1b78f541f028195a9c8"} Nov 03 20:25:21.182769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:21.180179 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89faeaab9182a587105783f4203c4e6999dd246d27c7d1b78f541f028195a9c8" Nov 03 20:25:21.182769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:21.180236 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7" Nov 03 20:25:21.186894 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:21.186863 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" event={"ID":"95b9f28c-e165-436f-a5a5-ae89d046297f","Type":"ContainerStarted","Data":"48d4a683a2677878859d64ba5446aad957d6f45333c86caab40d425fdbd1f530"} Nov 03 20:25:21.343129 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:25:21.343092 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b33145_a82b_4519_ab6a_c7ce0248992f.slice/crio-0fbdcc6abd5c7cb8a1505766b09e2cbb67de0dad9822c0c3f9a7f76661294ee5 WatchSource:0}: Error finding container 0fbdcc6abd5c7cb8a1505766b09e2cbb67de0dad9822c0c3f9a7f76661294ee5: Status 404 returned error can't find the container with id 0fbdcc6abd5c7cb8a1505766b09e2cbb67de0dad9822c0c3f9a7f76661294ee5 Nov 03 20:25:21.343273 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:21.343216 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn"] Nov 03 20:25:21.346137 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:21.346107 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:25:21.346226 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:21.346169 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:25:21.346226 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:21.346196 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:25:22.196107 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:22.196066 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad" path="/var/lib/kubelet/pods/97b5ec9f-f8a2-4b80-b1ca-e405a46aa8ad/volumes" Nov 03 20:25:22.199029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:22.199002 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" event={"ID":"d5b33145-a82b-4519-ab6a-c7ce0248992f","Type":"ContainerStarted","Data":"ad67ec99016939ed0b81e2b594857c37cd515de92a6bfdd357f477b7662cbc9a"} Nov 03 20:25:22.199162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:22.199049 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" event={"ID":"d5b33145-a82b-4519-ab6a-c7ce0248992f","Type":"ContainerStarted","Data":"0fbdcc6abd5c7cb8a1505766b09e2cbb67de0dad9822c0c3f9a7f76661294ee5"} Nov 03 20:25:22.200864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:22.200842 5111 generic.go:356] "Generic (PLEG): container finished" podID="95b9f28c-e165-436f-a5a5-ae89d046297f" containerID="48d4a683a2677878859d64ba5446aad957d6f45333c86caab40d425fdbd1f530" exitCode=0 Nov 03 20:25:22.200974 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:22.200894 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" event={"ID":"95b9f28c-e165-436f-a5a5-ae89d046297f","Type":"ContainerDied","Data":"48d4a683a2677878859d64ba5446aad957d6f45333c86caab40d425fdbd1f530"} Nov 03 20:25:22.220027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:22.219971 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" podStartSLOduration=2.219955238 podStartE2EDuration="2.219955238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:22.21610652 +0000 UTC m=+2752.178484958" watchObservedRunningTime="2026-11-03 20:25:22.219955238 +0000 UTC m=+2752.182333677" Nov 03 20:25:22.490340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:22.490215 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" Nov 03 20:25:22.535181 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:22.535144 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b9f28c-e165-436f-a5a5-ae89d046297f-kube-api-access-gz5w6\" (UniqueName: \"kubernetes.io/projected/95b9f28c-e165-436f-a5a5-ae89d046297f-kube-api-access-gz5w6\") pod \"95b9f28c-e165-436f-a5a5-ae89d046297f\" (UID: \"95b9f28c-e165-436f-a5a5-ae89d046297f\") " Nov 03 20:25:22.535341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:22.535226 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b9f28c-e165-436f-a5a5-ae89d046297f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b9f28c-e165-436f-a5a5-ae89d046297f-maas-api-openshift-ca\") pod \"95b9f28c-e165-436f-a5a5-ae89d046297f\" (UID: \"95b9f28c-e165-436f-a5a5-ae89d046297f\") " Nov 03 20:25:22.535341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:22.535289 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95b9f28c-e165-436f-a5a5-ae89d046297f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b9f28c-e165-436f-a5a5-ae89d046297f-maas-api-xks-ca\") pod \"95b9f28c-e165-436f-a5a5-ae89d046297f\" (UID: \"95b9f28c-e165-436f-a5a5-ae89d046297f\") " Nov 03 20:25:22.536512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:22.536488 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b9f28c-e165-436f-a5a5-ae89d046297f-maas-api-openshift-ca" pod "95b9f28c-e165-436f-a5a5-ae89d046297f" (UID: "95b9f28c-e165-436f-a5a5-ae89d046297f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:22.538191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:22.538168 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b9f28c-e165-436f-a5a5-ae89d046297f-kube-api-access-gz5w6" pod "95b9f28c-e165-436f-a5a5-ae89d046297f" (UID: "95b9f28c-e165-436f-a5a5-ae89d046297f"). InnerVolumeSpecName "kube-api-access-gz5w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:22.541300 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:22.541272 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b9f28c-e165-436f-a5a5-ae89d046297f-maas-api-xks-ca" pod "95b9f28c-e165-436f-a5a5-ae89d046297f" (UID: "95b9f28c-e165-436f-a5a5-ae89d046297f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:22.703567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:22.701966 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz5w6\" (UniqueName: \"kubernetes.io/projected/95b9f28c-e165-436f-a5a5-ae89d046297f-kube-api-access-gz5w6\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:22.703567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:22.702006 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b9f28c-e165-436f-a5a5-ae89d046297f-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:22.703567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:22.702021 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b9f28c-e165-436f-a5a5-ae89d046297f-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:23.031259 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:23.031215 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:23.032809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:23.032783 5111 patch_prober.go:29] interesting pod/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Nov 03 20:25:23.032944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:23.032833 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" podUID="d5b33145-a82b-4519-ab6a-c7ce0248992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Nov 03 20:25:23.214610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:23.214565 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" event={"ID":"95b9f28c-e165-436f-a5a5-ae89d046297f","Type":"ContainerDied","Data":"1ae541a12934e19747bf0e3a77e51bada5a586f56b6e977ecaf29bca022cc42c"} Nov 03 20:25:23.215111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:23.214609 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8" Nov 03 20:25:23.215111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:23.214633 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ae541a12934e19747bf0e3a77e51bada5a586f56b6e977ecaf29bca022cc42c" Nov 03 20:25:24.031410 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:24.031368 5111 patch_prober.go:29] interesting pod/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Nov 03 20:25:24.031688 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:24.031431 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" podUID="d5b33145-a82b-4519-ab6a-c7ce0248992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Nov 03 20:25:25.051564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:25.032232 5111 patch_prober.go:29] interesting pod/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Nov 03 20:25:25.051564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:25.032307 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" podUID="d5b33145-a82b-4519-ab6a-c7ce0248992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Nov 03 20:25:26.032903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:26.032402 5111 patch_prober.go:29] interesting pod/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Nov 03 20:25:26.032903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:26.032552 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" podUID="d5b33145-a82b-4519-ab6a-c7ce0248992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Nov 03 20:25:27.031508 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:27.031465 5111 patch_prober.go:29] interesting pod/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Nov 03 20:25:27.032052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:27.031568 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" podUID="d5b33145-a82b-4519-ab6a-c7ce0248992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Nov 03 20:25:28.032375 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:28.032336 5111 patch_prober.go:29] interesting pod/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Nov 03 20:25:28.032845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:28.032400 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" podUID="d5b33145-a82b-4519-ab6a-c7ce0248992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Nov 03 20:25:29.051583 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.049019 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:29.051583 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.050448 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:29.062822 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.062770 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:25:29.386605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.386333 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4"] Nov 03 20:25:29.408571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.387639 5111 cpu_manager.go:414] "containerMap: removing container" podUID="95b9f28c-e165-436f-a5a5-ae89d046297f" containerName="revoke-keys" Nov 03 20:25:29.408571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.400620 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b9f28c-e165-436f-a5a5-ae89d046297f" containerName="revoke-keys" Nov 03 20:25:29.408571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.400904 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b9f28c-e165-436f-a5a5-ae89d046297f" containerName="revoke-keys" Nov 03 20:25:29.521558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.510727 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4" Nov 03 20:25:29.554575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.537978 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:25:29.620557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.591809 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db6cac85-00c7-479b-9de7-52c66c2749aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fed3e4e-spfv4\" (UID: \"db6cac85-00c7-479b-9de7-52c66c2749aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4" Nov 03 20:25:29.620557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.591901 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpx9g\" (UniqueName: \"kubernetes.io/projected/db6cac85-00c7-479b-9de7-52c66c2749aa-kube-api-access-zpx9g\") pod \"maas-api-revoke-sub-4fed3e4e-spfv4\" (UID: \"db6cac85-00c7-479b-9de7-52c66c2749aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4" Nov 03 20:25:29.620557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.592159 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db6cac85-00c7-479b-9de7-52c66c2749aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fed3e4e-spfv4\" (UID: \"db6cac85-00c7-479b-9de7-52c66c2749aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4" Nov 03 20:25:29.620557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.596938 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:25:29.671621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.669888 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4"] Nov 03 20:25:29.699423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.688365 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv"] Nov 03 20:25:29.699423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.695077 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db6cac85-00c7-479b-9de7-52c66c2749aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fed3e4e-spfv4\" (UID: \"db6cac85-00c7-479b-9de7-52c66c2749aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4" Nov 03 20:25:29.699423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.695134 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpx9g\" (UniqueName: \"kubernetes.io/projected/db6cac85-00c7-479b-9de7-52c66c2749aa-kube-api-access-zpx9g\") pod \"maas-api-revoke-sub-4fed3e4e-spfv4\" (UID: \"db6cac85-00c7-479b-9de7-52c66c2749aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4" Nov 03 20:25:29.699423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.695202 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db6cac85-00c7-479b-9de7-52c66c2749aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fed3e4e-spfv4\" (UID: \"db6cac85-00c7-479b-9de7-52c66c2749aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4" Nov 03 20:25:29.699423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.695211 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-46c34c-lf8pv"] Nov 03 20:25:29.699423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.696221 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db6cac85-00c7-479b-9de7-52c66c2749aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fed3e4e-spfv4\" (UID: \"db6cac85-00c7-479b-9de7-52c66c2749aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4" Nov 03 20:25:29.699423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.699062 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db6cac85-00c7-479b-9de7-52c66c2749aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fed3e4e-spfv4\" (UID: \"db6cac85-00c7-479b-9de7-52c66c2749aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4" Nov 03 20:25:29.707573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.707547 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpx9g\" (UniqueName: \"kubernetes.io/projected/db6cac85-00c7-479b-9de7-52c66c2749aa-kube-api-access-zpx9g\") pod \"maas-api-revoke-sub-4fed3e4e-spfv4\" (UID: \"db6cac85-00c7-479b-9de7-52c66c2749aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4" Nov 03 20:25:29.999607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:29.997896 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4" Nov 03 20:25:30.191332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:30.191293 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df9aa3f5-199d-4126-a993-39240f7a098c" path="/var/lib/kubelet/pods/df9aa3f5-199d-4126-a993-39240f7a098c/volumes" Nov 03 20:25:30.337318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:30.337290 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4"] Nov 03 20:25:30.337652 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:25:30.337625 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6cac85_00c7_479b_9de7_52c66c2749aa.slice/crio-23af8f79a56391a7ade320aac9eb3c8c68b2af0f49c5aa2d1f4912ea2d6915d8 WatchSource:0}: Error finding container 23af8f79a56391a7ade320aac9eb3c8c68b2af0f49c5aa2d1f4912ea2d6915d8: Status 404 returned error can't find the container with id 23af8f79a56391a7ade320aac9eb3c8c68b2af0f49c5aa2d1f4912ea2d6915d8 Nov 03 20:25:30.602518 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:30.602428 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4" event={"ID":"db6cac85-00c7-479b-9de7-52c66c2749aa","Type":"ContainerStarted","Data":"23af8f79a56391a7ade320aac9eb3c8c68b2af0f49c5aa2d1f4912ea2d6915d8"} Nov 03 20:25:31.612888 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:31.612806 5111 generic.go:356] "Generic (PLEG): container finished" podID="db6cac85-00c7-479b-9de7-52c66c2749aa" containerID="27885f6751f3869220e3a5ad8a31a8e7d0ff2cf8338357576c46d1a6db4587e7" exitCode=0 Nov 03 20:25:31.613321 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:31.612894 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4" event={"ID":"db6cac85-00c7-479b-9de7-52c66c2749aa","Type":"ContainerDied","Data":"27885f6751f3869220e3a5ad8a31a8e7d0ff2cf8338357576c46d1a6db4587e7"} Nov 03 20:25:32.810491 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:32.810461 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4" Nov 03 20:25:32.888415 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:32.888335 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db6cac85-00c7-479b-9de7-52c66c2749aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db6cac85-00c7-479b-9de7-52c66c2749aa-maas-api-xks-ca\") pod \"db6cac85-00c7-479b-9de7-52c66c2749aa\" (UID: \"db6cac85-00c7-479b-9de7-52c66c2749aa\") " Nov 03 20:25:32.889013 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:32.888985 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db6cac85-00c7-479b-9de7-52c66c2749aa-kube-api-access-zpx9g\" (UniqueName: \"kubernetes.io/projected/db6cac85-00c7-479b-9de7-52c66c2749aa-kube-api-access-zpx9g\") pod \"db6cac85-00c7-479b-9de7-52c66c2749aa\" (UID: \"db6cac85-00c7-479b-9de7-52c66c2749aa\") " Nov 03 20:25:32.889219 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:32.889200 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db6cac85-00c7-479b-9de7-52c66c2749aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db6cac85-00c7-479b-9de7-52c66c2749aa-maas-api-openshift-ca\") pod \"db6cac85-00c7-479b-9de7-52c66c2749aa\" (UID: \"db6cac85-00c7-479b-9de7-52c66c2749aa\") " Nov 03 20:25:32.889906 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:32.889877 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db6cac85-00c7-479b-9de7-52c66c2749aa-maas-api-openshift-ca" pod "db6cac85-00c7-479b-9de7-52c66c2749aa" (UID: "db6cac85-00c7-479b-9de7-52c66c2749aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:32.890418 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:32.890390 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db6cac85-00c7-479b-9de7-52c66c2749aa-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:32.892045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:32.891971 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db6cac85-00c7-479b-9de7-52c66c2749aa-maas-api-xks-ca" pod "db6cac85-00c7-479b-9de7-52c66c2749aa" (UID: "db6cac85-00c7-479b-9de7-52c66c2749aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:32.892811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:32.892790 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db6cac85-00c7-479b-9de7-52c66c2749aa-kube-api-access-zpx9g" pod "db6cac85-00c7-479b-9de7-52c66c2749aa" (UID: "db6cac85-00c7-479b-9de7-52c66c2749aa"). InnerVolumeSpecName "kube-api-access-zpx9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:33.048556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.039450 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db6cac85-00c7-479b-9de7-52c66c2749aa-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:33.048556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.039497 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpx9g\" (UniqueName: \"kubernetes.io/projected/db6cac85-00c7-479b-9de7-52c66c2749aa-kube-api-access-zpx9g\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:33.332660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.331522 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2"] Nov 03 20:25:33.335228 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.332883 5111 cpu_manager.go:414] "containerMap: removing container" podUID="db6cac85-00c7-479b-9de7-52c66c2749aa" containerName="revoke-keys" Nov 03 20:25:33.335228 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.332910 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db6cac85-00c7-479b-9de7-52c66c2749aa" containerName="revoke-keys" Nov 03 20:25:33.335228 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.333161 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="db6cac85-00c7-479b-9de7-52c66c2749aa" containerName="revoke-keys" Nov 03 20:25:33.360707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.360671 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2"] Nov 03 20:25:33.361039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.360807 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" Nov 03 20:25:33.474517 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.474460 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tpcq\" (UniqueName: \"kubernetes.io/projected/5ca18671-d98f-405b-b297-6a0b09a217be-kube-api-access-2tpcq\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2\" (UID: \"5ca18671-d98f-405b-b297-6a0b09a217be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" Nov 03 20:25:33.474781 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.474623 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca18671-d98f-405b-b297-6a0b09a217be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2\" (UID: \"5ca18671-d98f-405b-b297-6a0b09a217be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" Nov 03 20:25:33.474781 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.474745 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca18671-d98f-405b-b297-6a0b09a217be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2\" (UID: \"5ca18671-d98f-405b-b297-6a0b09a217be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" Nov 03 20:25:33.576759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.576721 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca18671-d98f-405b-b297-6a0b09a217be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2\" (UID: \"5ca18671-d98f-405b-b297-6a0b09a217be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" Nov 03 20:25:33.576977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.576806 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca18671-d98f-405b-b297-6a0b09a217be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2\" (UID: \"5ca18671-d98f-405b-b297-6a0b09a217be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" Nov 03 20:25:33.577089 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.577040 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tpcq\" (UniqueName: \"kubernetes.io/projected/5ca18671-d98f-405b-b297-6a0b09a217be-kube-api-access-2tpcq\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2\" (UID: \"5ca18671-d98f-405b-b297-6a0b09a217be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" Nov 03 20:25:33.577580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.577523 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca18671-d98f-405b-b297-6a0b09a217be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2\" (UID: \"5ca18671-d98f-405b-b297-6a0b09a217be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" Nov 03 20:25:33.579401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.579375 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca18671-d98f-405b-b297-6a0b09a217be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2\" (UID: \"5ca18671-d98f-405b-b297-6a0b09a217be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" Nov 03 20:25:33.584942 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.584887 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tpcq\" (UniqueName: \"kubernetes.io/projected/5ca18671-d98f-405b-b297-6a0b09a217be-kube-api-access-2tpcq\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2\" (UID: \"5ca18671-d98f-405b-b297-6a0b09a217be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" Nov 03 20:25:33.633155 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.633124 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4" Nov 03 20:25:33.633359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.633153 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4" event={"ID":"db6cac85-00c7-479b-9de7-52c66c2749aa","Type":"ContainerDied","Data":"23af8f79a56391a7ade320aac9eb3c8c68b2af0f49c5aa2d1f4912ea2d6915d8"} Nov 03 20:25:33.633359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.633203 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23af8f79a56391a7ade320aac9eb3c8c68b2af0f49c5aa2d1f4912ea2d6915d8" Nov 03 20:25:33.694095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.694064 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" Nov 03 20:25:33.988276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:33.987664 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2"] Nov 03 20:25:33.988760 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:25:33.988332 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca18671_d98f_405b_b297_6a0b09a217be.slice/crio-bef6fb641d033483aa96f0644f9312eadd9bcbf7dd8062c2feefd4302aa50ce2 WatchSource:0}: Error finding container bef6fb641d033483aa96f0644f9312eadd9bcbf7dd8062c2feefd4302aa50ce2: Status 404 returned error can't find the container with id bef6fb641d033483aa96f0644f9312eadd9bcbf7dd8062c2feefd4302aa50ce2 Nov 03 20:25:34.643995 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:34.643952 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" event={"ID":"5ca18671-d98f-405b-b297-6a0b09a217be","Type":"ContainerStarted","Data":"185c24c4cd92518ae95dd98b63640701767bae3f5744031b59db94308a55fc1b"} Nov 03 20:25:34.644153 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:34.644033 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" event={"ID":"5ca18671-d98f-405b-b297-6a0b09a217be","Type":"ContainerStarted","Data":"bef6fb641d033483aa96f0644f9312eadd9bcbf7dd8062c2feefd4302aa50ce2"} Nov 03 20:25:34.664209 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:34.664159 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" podStartSLOduration=1.664145478 podStartE2EDuration="1.664145478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:34.658518755 +0000 UTC m=+2764.620897203" watchObservedRunningTime="2026-11-03 20:25:34.664145478 +0000 UTC m=+2764.626523950" Nov 03 20:25:35.061018 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:25:35.060935 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:25:35.655280 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:35.655244 5111 generic.go:356] "Generic (PLEG): container finished" podID="5ca18671-d98f-405b-b297-6a0b09a217be" containerID="185c24c4cd92518ae95dd98b63640701767bae3f5744031b59db94308a55fc1b" exitCode=0 Nov 03 20:25:35.655492 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:35.655326 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" event={"ID":"5ca18671-d98f-405b-b297-6a0b09a217be","Type":"ContainerDied","Data":"185c24c4cd92518ae95dd98b63640701767bae3f5744031b59db94308a55fc1b"} Nov 03 20:25:36.231111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:36.231072 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl"] Nov 03 20:25:36.231651 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:36.231324 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" podUID="8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" containerName="istio-proxy" containerID="cri-o://dd41eef0cff8d6939e3a40aa2c2a1611017397716eb156b3af0ff32a20b29fd8" gracePeriod=30 Nov 03 20:25:36.239738 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:36.239117 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn"] Nov 03 20:25:36.243872 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:36.240392 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" podUID="b7fc530f-7898-450f-bc05-be85b74215cf" containerName="istio-proxy" containerID="cri-o://e075fee0e93ed3df9fc2733679b7f3e252cbb027768ec90c3872917b3a13ca05" gracePeriod=30 Nov 03 20:25:36.896606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:36.896571 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd"] Nov 03 20:25:36.902271 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:36.902243 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd" Nov 03 20:25:36.918588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:36.918551 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd"] Nov 03 20:25:36.972798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:36.967455 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd\" (UID: \"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd" Nov 03 20:25:36.972798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:36.967623 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwvk2\" (UniqueName: \"kubernetes.io/projected/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-kube-api-access-gwvk2\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd\" (UID: \"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd" Nov 03 20:25:36.972798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:36.967672 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd\" (UID: \"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd" Nov 03 20:25:37.069029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.068989 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd\" (UID: \"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd" Nov 03 20:25:37.069268 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.069254 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd\" (UID: \"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd" Nov 03 20:25:37.069375 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.069362 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwvk2\" (UniqueName: \"kubernetes.io/projected/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-kube-api-access-gwvk2\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd\" (UID: \"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd" Nov 03 20:25:37.069961 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.069934 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd\" (UID: \"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd" Nov 03 20:25:37.071692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.071670 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd\" (UID: \"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd" Nov 03 20:25:37.076583 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.076085 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" Nov 03 20:25:37.077468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.077448 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwvk2\" (UniqueName: \"kubernetes.io/projected/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-kube-api-access-gwvk2\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd\" (UID: \"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd" Nov 03 20:25:37.171205 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.171165 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca18671-d98f-405b-b297-6a0b09a217be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca18671-d98f-405b-b297-6a0b09a217be-maas-api-openshift-ca\") pod \"5ca18671-d98f-405b-b297-6a0b09a217be\" (UID: \"5ca18671-d98f-405b-b297-6a0b09a217be\") " Nov 03 20:25:37.171394 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.171251 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca18671-d98f-405b-b297-6a0b09a217be-kube-api-access-2tpcq\" (UniqueName: \"kubernetes.io/projected/5ca18671-d98f-405b-b297-6a0b09a217be-kube-api-access-2tpcq\") pod \"5ca18671-d98f-405b-b297-6a0b09a217be\" (UID: \"5ca18671-d98f-405b-b297-6a0b09a217be\") " Nov 03 20:25:37.171394 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.171353 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ca18671-d98f-405b-b297-6a0b09a217be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca18671-d98f-405b-b297-6a0b09a217be-maas-api-xks-ca\") pod \"5ca18671-d98f-405b-b297-6a0b09a217be\" (UID: \"5ca18671-d98f-405b-b297-6a0b09a217be\") " Nov 03 20:25:37.171630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.171602 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca18671-d98f-405b-b297-6a0b09a217be-maas-api-openshift-ca" pod "5ca18671-d98f-405b-b297-6a0b09a217be" (UID: "5ca18671-d98f-405b-b297-6a0b09a217be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:37.172306 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.172279 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca18671-d98f-405b-b297-6a0b09a217be-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:37.173851 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.173825 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ca18671-d98f-405b-b297-6a0b09a217be-maas-api-xks-ca" pod "5ca18671-d98f-405b-b297-6a0b09a217be" (UID: "5ca18671-d98f-405b-b297-6a0b09a217be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:37.173971 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.173882 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca18671-d98f-405b-b297-6a0b09a217be-kube-api-access-2tpcq" pod "5ca18671-d98f-405b-b297-6a0b09a217be" (UID: "5ca18671-d98f-405b-b297-6a0b09a217be"). InnerVolumeSpecName "kube-api-access-2tpcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:37.222078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.222028 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd" Nov 03 20:25:37.275555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.275505 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tpcq\" (UniqueName: \"kubernetes.io/projected/5ca18671-d98f-405b-b297-6a0b09a217be-kube-api-access-2tpcq\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:37.276017 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.275985 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca18671-d98f-405b-b297-6a0b09a217be-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:37.400833 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.389873 5111 scope.go:122] "RemoveContainer" containerID="1f17e5e23d97be66acb5b2f2819929f33fbd2cc4c1332bd8a6ec5d06deb84886" Nov 03 20:25:37.411921 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.411893 5111 scope.go:122] "RemoveContainer" containerID="bfa979a1e7b060995ab0aab437fd27b8feb3355901582b3d6d7ad3a6e4636f15" Nov 03 20:25:37.514664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.514613 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd"] Nov 03 20:25:37.514968 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:25:37.514932 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac5fa53_a2ee_4811_8eff_fbd745e00a7d.slice/crio-881cac1a4482b340bdfc1965270c3434ab7c7b4327a17972fb3df027534d1743 WatchSource:0}: Error finding container 881cac1a4482b340bdfc1965270c3434ab7c7b4327a17972fb3df027534d1743: Status 404 returned error can't find the container with id 881cac1a4482b340bdfc1965270c3434ab7c7b4327a17972fb3df027534d1743 Nov 03 20:25:37.857292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.857260 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" Nov 03 20:25:37.857292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.857289 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2" event={"ID":"5ca18671-d98f-405b-b297-6a0b09a217be","Type":"ContainerDied","Data":"bef6fb641d033483aa96f0644f9312eadd9bcbf7dd8062c2feefd4302aa50ce2"} Nov 03 20:25:37.857549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.857342 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bef6fb641d033483aa96f0644f9312eadd9bcbf7dd8062c2feefd4302aa50ce2" Nov 03 20:25:37.858874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:37.858842 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd" event={"ID":"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d","Type":"ContainerStarted","Data":"881cac1a4482b340bdfc1965270c3434ab7c7b4327a17972fb3df027534d1743"} Nov 03 20:25:38.874129 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:38.874090 5111 generic.go:356] "Generic (PLEG): container finished" podID="6ac5fa53-a2ee-4811-8eff-fbd745e00a7d" containerID="fdfafd3471c520281796e3f08b43fa222014e8d43ab79bd06fd1e920c59e7049" exitCode=0 Nov 03 20:25:38.874588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:38.874135 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd" event={"ID":"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d","Type":"ContainerDied","Data":"fdfafd3471c520281796e3f08b43fa222014e8d43ab79bd06fd1e920c59e7049"} Nov 03 20:25:40.068165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:40.068137 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd" Nov 03 20:25:40.156066 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:40.155960 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-maas-api-xks-ca\") pod \"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d\" (UID: \"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d\") " Nov 03 20:25:40.156244 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:40.156120 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-kube-api-access-gwvk2\" (UniqueName: \"kubernetes.io/projected/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-kube-api-access-gwvk2\") pod \"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d\" (UID: \"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d\") " Nov 03 20:25:40.156244 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:40.156211 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-maas-api-openshift-ca\") pod \"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d\" (UID: \"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d\") " Nov 03 20:25:40.156653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:40.156612 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-maas-api-openshift-ca" pod "6ac5fa53-a2ee-4811-8eff-fbd745e00a7d" (UID: "6ac5fa53-a2ee-4811-8eff-fbd745e00a7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:40.157069 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:40.157043 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:40.158186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:40.158164 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-maas-api-xks-ca" pod "6ac5fa53-a2ee-4811-8eff-fbd745e00a7d" (UID: "6ac5fa53-a2ee-4811-8eff-fbd745e00a7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:40.158322 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:40.158303 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-kube-api-access-gwvk2" pod "6ac5fa53-a2ee-4811-8eff-fbd745e00a7d" (UID: "6ac5fa53-a2ee-4811-8eff-fbd745e00a7d"). InnerVolumeSpecName "kube-api-access-gwvk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:40.259525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:40.259484 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwvk2\" (UniqueName: \"kubernetes.io/projected/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-kube-api-access-gwvk2\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:40.259525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:40.259522 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:40.894145 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:40.894116 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd" Nov 03 20:25:40.894325 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:40.894147 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd" event={"ID":"6ac5fa53-a2ee-4811-8eff-fbd745e00a7d","Type":"ContainerDied","Data":"881cac1a4482b340bdfc1965270c3434ab7c7b4327a17972fb3df027534d1743"} Nov 03 20:25:40.894325 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:40.894194 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="881cac1a4482b340bdfc1965270c3434ab7c7b4327a17972fb3df027534d1743" Nov 03 20:25:41.325761 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.325727 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq"] Nov 03 20:25:41.326481 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.326467 5111 cpu_manager.go:414] "containerMap: removing container" podUID="5ca18671-d98f-405b-b297-6a0b09a217be" containerName="revoke-keys" Nov 03 20:25:41.326550 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.326483 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca18671-d98f-405b-b297-6a0b09a217be" containerName="revoke-keys" Nov 03 20:25:41.326596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.326589 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6ac5fa53-a2ee-4811-8eff-fbd745e00a7d" containerName="revoke-keys" Nov 03 20:25:41.326658 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.326596 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac5fa53-a2ee-4811-8eff-fbd745e00a7d" containerName="revoke-keys" Nov 03 20:25:41.326754 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.326743 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca18671-d98f-405b-b297-6a0b09a217be" containerName="revoke-keys" Nov 03 20:25:41.326790 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.326768 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac5fa53-a2ee-4811-8eff-fbd745e00a7d" containerName="revoke-keys" Nov 03 20:25:41.330798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.330775 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" Nov 03 20:25:41.335260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.334809 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:25:41.335260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.334933 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:25:41.340192 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.340166 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq"] Nov 03 20:25:41.501123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.501041 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d30c61-40a8-4894-b554-1f4d75916150-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq\" (UID: \"11d30c61-40a8-4894-b554-1f4d75916150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" Nov 03 20:25:41.501325 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.501157 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d30c61-40a8-4894-b554-1f4d75916150-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq\" (UID: \"11d30c61-40a8-4894-b554-1f4d75916150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" Nov 03 20:25:41.501325 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.501266 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz45p\" (UniqueName: \"kubernetes.io/projected/11d30c61-40a8-4894-b554-1f4d75916150-kube-api-access-vz45p\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq\" (UID: \"11d30c61-40a8-4894-b554-1f4d75916150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" Nov 03 20:25:41.599807 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.599780 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:25:41.604018 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.603980 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz45p\" (UniqueName: \"kubernetes.io/projected/11d30c61-40a8-4894-b554-1f4d75916150-kube-api-access-vz45p\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq\" (UID: \"11d30c61-40a8-4894-b554-1f4d75916150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" Nov 03 20:25:41.604385 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.604363 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d30c61-40a8-4894-b554-1f4d75916150-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq\" (UID: \"11d30c61-40a8-4894-b554-1f4d75916150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" Nov 03 20:25:41.604502 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.604429 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d30c61-40a8-4894-b554-1f4d75916150-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq\" (UID: \"11d30c61-40a8-4894-b554-1f4d75916150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" Nov 03 20:25:41.606090 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.606063 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d30c61-40a8-4894-b554-1f4d75916150-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq\" (UID: \"11d30c61-40a8-4894-b554-1f4d75916150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" Nov 03 20:25:41.607852 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.607817 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d30c61-40a8-4894-b554-1f4d75916150-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq\" (UID: \"11d30c61-40a8-4894-b554-1f4d75916150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" Nov 03 20:25:41.612687 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.612655 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz45p\" (UniqueName: \"kubernetes.io/projected/11d30c61-40a8-4894-b554-1f4d75916150-kube-api-access-vz45p\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq\" (UID: \"11d30c61-40a8-4894-b554-1f4d75916150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" Nov 03 20:25:41.640037 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.640008 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:25:41.705088 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.705052 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-istio-envoy\") pod \"b7fc530f-7898-450f-bc05-be85b74215cf\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " Nov 03 20:25:41.705263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.705202 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-credential-socket\") pod \"b7fc530f-7898-450f-bc05-be85b74215cf\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " Nov 03 20:25:41.705263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.705253 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-workload-certs\") pod \"b7fc530f-7898-450f-bc05-be85b74215cf\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " Nov 03 20:25:41.705339 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.705301 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7fc530f-7898-450f-bc05-be85b74215cf-istio-token\" (UniqueName: \"kubernetes.io/projected/b7fc530f-7898-450f-bc05-be85b74215cf-istio-token\") pod \"b7fc530f-7898-450f-bc05-be85b74215cf\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " Nov 03 20:25:41.705376 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.705348 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-istio-data\") pod \"b7fc530f-7898-450f-bc05-be85b74215cf\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " Nov 03 20:25:41.705601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.705470 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-credential-socket" pod "b7fc530f-7898-450f-bc05-be85b74215cf" (UID: "b7fc530f-7898-450f-bc05-be85b74215cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:41.705715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.705662 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-workload-socket\") pod \"b7fc530f-7898-450f-bc05-be85b74215cf\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " Nov 03 20:25:41.705715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.705699 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7fc530f-7898-450f-bc05-be85b74215cf-kube-api-access-cwps5\" (UniqueName: \"kubernetes.io/projected/b7fc530f-7898-450f-bc05-be85b74215cf-kube-api-access-cwps5\") pod \"b7fc530f-7898-450f-bc05-be85b74215cf\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " Nov 03 20:25:41.705844 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.705809 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7fc530f-7898-450f-bc05-be85b74215cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7fc530f-7898-450f-bc05-be85b74215cf-istio-podinfo\") pod \"b7fc530f-7898-450f-bc05-be85b74215cf\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " Nov 03 20:25:41.705900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.705849 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-workload-certs" pod "b7fc530f-7898-450f-bc05-be85b74215cf" (UID: "b7fc530f-7898-450f-bc05-be85b74215cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:41.705900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.705857 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7fc530f-7898-450f-bc05-be85b74215cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7fc530f-7898-450f-bc05-be85b74215cf-istiod-ca-cert\") pod \"b7fc530f-7898-450f-bc05-be85b74215cf\" (UID: \"b7fc530f-7898-450f-bc05-be85b74215cf\") " Nov 03 20:25:41.706414 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.706171 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7fc530f-7898-450f-bc05-be85b74215cf-istiod-ca-cert" pod "b7fc530f-7898-450f-bc05-be85b74215cf" (UID: "b7fc530f-7898-450f-bc05-be85b74215cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:41.706414 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.706378 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-workload-socket" pod "b7fc530f-7898-450f-bc05-be85b74215cf" (UID: "b7fc530f-7898-450f-bc05-be85b74215cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:41.706652 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.706449 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-istio-data" pod "b7fc530f-7898-450f-bc05-be85b74215cf" (UID: "b7fc530f-7898-450f-bc05-be85b74215cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:41.706863 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.706833 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.706940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.706863 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.706940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.706877 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.706940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.706891 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.706940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.706905 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7fc530f-7898-450f-bc05-be85b74215cf-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.708346 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.708317 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-istio-envoy" pod "b7fc530f-7898-450f-bc05-be85b74215cf" (UID: "b7fc530f-7898-450f-bc05-be85b74215cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:41.708950 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.708913 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7fc530f-7898-450f-bc05-be85b74215cf-istio-token" pod "b7fc530f-7898-450f-bc05-be85b74215cf" (UID: "b7fc530f-7898-450f-bc05-be85b74215cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:41.709101 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.709066 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7fc530f-7898-450f-bc05-be85b74215cf-kube-api-access-cwps5" pod "b7fc530f-7898-450f-bc05-be85b74215cf" (UID: "b7fc530f-7898-450f-bc05-be85b74215cf"). InnerVolumeSpecName "kube-api-access-cwps5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:41.709454 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.709432 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7fc530f-7898-450f-bc05-be85b74215cf-istio-podinfo" pod "b7fc530f-7898-450f-bc05-be85b74215cf" (UID: "b7fc530f-7898-450f-bc05-be85b74215cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:25:41.731742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.731711 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" Nov 03 20:25:41.813315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.808155 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-kube-api-access-mbrbt\" (UniqueName: \"kubernetes.io/projected/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-kube-api-access-mbrbt\") pod \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " Nov 03 20:25:41.813315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.808209 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-data\") pod \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " Nov 03 20:25:41.813315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.808342 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-credential-socket\") pod \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " Nov 03 20:25:41.813315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.808386 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-token\" (UniqueName: \"kubernetes.io/projected/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-token\") pod \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " Nov 03 20:25:41.813315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.808444 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istiod-ca-cert\") pod \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " Nov 03 20:25:41.813315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.808608 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-podinfo\") pod \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " Nov 03 20:25:41.813315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.808703 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-envoy\") pod \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " Nov 03 20:25:41.813315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.808774 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-workload-certs\") pod \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " Nov 03 20:25:41.813315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.808894 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-workload-socket\") pod \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\" (UID: \"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7\") " Nov 03 20:25:41.813315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.809873 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7fc530f-7898-450f-bc05-be85b74215cf-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.813315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.809897 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwps5\" (UniqueName: \"kubernetes.io/projected/b7fc530f-7898-450f-bc05-be85b74215cf-kube-api-access-cwps5\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.813315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.809912 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7fc530f-7898-450f-bc05-be85b74215cf-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.813315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.809927 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7fc530f-7898-450f-bc05-be85b74215cf-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.813315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.810219 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-workload-socket" pod "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" (UID: "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:41.813315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.812978 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-kube-api-access-mbrbt" pod "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" (UID: "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7"). InnerVolumeSpecName "kube-api-access-mbrbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:41.813315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.813180 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-data" pod "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" (UID: "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:41.814186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.813757 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-workload-certs" pod "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" (UID: "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:41.814186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.813859 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istiod-ca-cert" pod "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" (UID: "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:41.814186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.813957 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-credential-socket" pod "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" (UID: "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:41.815719 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.815687 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-podinfo" pod "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" (UID: "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:25:41.816223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.816190 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-envoy" pod "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" (UID: "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:41.816382 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.816357 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-token" pod "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" (UID: "8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:41.904242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.904209 5111 generic.go:356] "Generic (PLEG): container finished" podID="8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" containerID="dd41eef0cff8d6939e3a40aa2c2a1611017397716eb156b3af0ff32a20b29fd8" exitCode=0 Nov 03 20:25:41.904435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.904259 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" Nov 03 20:25:41.904435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.904275 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" event={"ID":"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7","Type":"ContainerDied","Data":"dd41eef0cff8d6939e3a40aa2c2a1611017397716eb156b3af0ff32a20b29fd8"} Nov 03 20:25:41.904435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.904333 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl" event={"ID":"8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7","Type":"ContainerDied","Data":"8b328fc7e8f43d8139ffe5ccbe737247dfa5e3e0c1db6283897e930d9755cc62"} Nov 03 20:25:41.904435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.904349 5111 scope.go:122] "RemoveContainer" containerID="dd41eef0cff8d6939e3a40aa2c2a1611017397716eb156b3af0ff32a20b29fd8" Nov 03 20:25:41.906002 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.905973 5111 generic.go:356] "Generic (PLEG): container finished" podID="b7fc530f-7898-450f-bc05-be85b74215cf" containerID="e075fee0e93ed3df9fc2733679b7f3e252cbb027768ec90c3872917b3a13ca05" exitCode=0 Nov 03 20:25:41.906138 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.906020 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" Nov 03 20:25:41.906138 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.906045 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" event={"ID":"b7fc530f-7898-450f-bc05-be85b74215cf","Type":"ContainerDied","Data":"e075fee0e93ed3df9fc2733679b7f3e252cbb027768ec90c3872917b3a13ca05"} Nov 03 20:25:41.906138 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.906087 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn" event={"ID":"b7fc530f-7898-450f-bc05-be85b74215cf","Type":"ContainerDied","Data":"615746a7cd777027cc86233c54c87b5e4fbdc0d47615741f4b23f02da8b28920"} Nov 03 20:25:41.911460 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.911433 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.911577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.911464 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.911577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.911477 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.911577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.911491 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.911577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.911503 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.911577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.911516 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.911577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.911543 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.911577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.911558 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbrbt\" (UniqueName: \"kubernetes.io/projected/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-kube-api-access-mbrbt\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.911577 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.911574 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:41.918505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.918483 5111 scope.go:122] "RemoveContainer" containerID="dd41eef0cff8d6939e3a40aa2c2a1611017397716eb156b3af0ff32a20b29fd8" Nov 03 20:25:41.918841 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:25:41.918819 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd41eef0cff8d6939e3a40aa2c2a1611017397716eb156b3af0ff32a20b29fd8\": container with ID starting with dd41eef0cff8d6939e3a40aa2c2a1611017397716eb156b3af0ff32a20b29fd8 not found: ID does not exist" containerID="dd41eef0cff8d6939e3a40aa2c2a1611017397716eb156b3af0ff32a20b29fd8" Nov 03 20:25:41.918930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.918851 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd41eef0cff8d6939e3a40aa2c2a1611017397716eb156b3af0ff32a20b29fd8"} err="failed to get container status \"dd41eef0cff8d6939e3a40aa2c2a1611017397716eb156b3af0ff32a20b29fd8\": rpc error: code = NotFound desc = could not find container \"dd41eef0cff8d6939e3a40aa2c2a1611017397716eb156b3af0ff32a20b29fd8\": container with ID starting with dd41eef0cff8d6939e3a40aa2c2a1611017397716eb156b3af0ff32a20b29fd8 not found: ID does not exist" Nov 03 20:25:41.918930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.918875 5111 scope.go:122] "RemoveContainer" containerID="e075fee0e93ed3df9fc2733679b7f3e252cbb027768ec90c3872917b3a13ca05" Nov 03 20:25:41.932417 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.932395 5111 scope.go:122] "RemoveContainer" containerID="e075fee0e93ed3df9fc2733679b7f3e252cbb027768ec90c3872917b3a13ca05" Nov 03 20:25:41.932850 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:25:41.932822 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e075fee0e93ed3df9fc2733679b7f3e252cbb027768ec90c3872917b3a13ca05\": container with ID starting with e075fee0e93ed3df9fc2733679b7f3e252cbb027768ec90c3872917b3a13ca05 not found: ID does not exist" containerID="e075fee0e93ed3df9fc2733679b7f3e252cbb027768ec90c3872917b3a13ca05" Nov 03 20:25:41.933005 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.932876 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e075fee0e93ed3df9fc2733679b7f3e252cbb027768ec90c3872917b3a13ca05"} err="failed to get container status \"e075fee0e93ed3df9fc2733679b7f3e252cbb027768ec90c3872917b3a13ca05\": rpc error: code = NotFound desc = could not find container \"e075fee0e93ed3df9fc2733679b7f3e252cbb027768ec90c3872917b3a13ca05\": container with ID starting with e075fee0e93ed3df9fc2733679b7f3e252cbb027768ec90c3872917b3a13ca05 not found: ID does not exist" Nov 03 20:25:41.946318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.946290 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl"] Nov 03 20:25:41.952981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.952954 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-q96rl"] Nov 03 20:25:41.958315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.958294 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn"] Nov 03 20:25:41.962489 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:41.962468 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-w5ndn"] Nov 03 20:25:42.022953 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:25:42.022919 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d30c61_40a8_4894_b554_1f4d75916150.slice/crio-5f84d573fca40e39fca7ab098d9c3af33e183cc0a98a861bb2946386a6661f63 WatchSource:0}: Error finding container 5f84d573fca40e39fca7ab098d9c3af33e183cc0a98a861bb2946386a6661f63: Status 404 returned error can't find the container with id 5f84d573fca40e39fca7ab098d9c3af33e183cc0a98a861bb2946386a6661f63 Nov 03 20:25:42.023129 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:42.023104 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq"] Nov 03 20:25:42.248558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:42.247771 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" path="/var/lib/kubelet/pods/8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7/volumes" Nov 03 20:25:42.258553 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:42.248769 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7fc530f-7898-450f-bc05-be85b74215cf" path="/var/lib/kubelet/pods/b7fc530f-7898-450f-bc05-be85b74215cf/volumes" Nov 03 20:25:42.971575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:42.968819 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" event={"ID":"11d30c61-40a8-4894-b554-1f4d75916150","Type":"ContainerStarted","Data":"4d6eef0d4fe65c38a2ad67f4524e7db56912fb6c4fc52eed9a875ac67776862b"} Nov 03 20:25:42.971575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:42.968895 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" event={"ID":"11d30c61-40a8-4894-b554-1f4d75916150","Type":"ContainerStarted","Data":"5f84d573fca40e39fca7ab098d9c3af33e183cc0a98a861bb2946386a6661f63"} Nov 03 20:25:43.054775 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:43.054702 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" podStartSLOduration=2.054679612 podStartE2EDuration="2.054679612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:43.006438385 +0000 UTC m=+2772.968816824" watchObservedRunningTime="2026-11-03 20:25:43.054679612 +0000 UTC m=+2773.017058052" Nov 03 20:25:44.084312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:44.084272 5111 generic.go:356] "Generic (PLEG): container finished" podID="11d30c61-40a8-4894-b554-1f4d75916150" containerID="4d6eef0d4fe65c38a2ad67f4524e7db56912fb6c4fc52eed9a875ac67776862b" exitCode=0 Nov 03 20:25:44.086022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:44.085990 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" event={"ID":"11d30c61-40a8-4894-b554-1f4d75916150","Type":"ContainerDied","Data":"4d6eef0d4fe65c38a2ad67f4524e7db56912fb6c4fc52eed9a875ac67776862b"} Nov 03 20:25:45.620152 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:45.620104 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2"] Nov 03 20:25:45.630337 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:45.630274 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ee2323-twzh2"] Nov 03 20:25:45.825556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:45.823147 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" Nov 03 20:25:45.970522 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:45.970437 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11d30c61-40a8-4894-b554-1f4d75916150-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d30c61-40a8-4894-b554-1f4d75916150-maas-api-xks-ca\") pod \"11d30c61-40a8-4894-b554-1f4d75916150\" (UID: \"11d30c61-40a8-4894-b554-1f4d75916150\") " Nov 03 20:25:45.970522 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:45.970496 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d30c61-40a8-4894-b554-1f4d75916150-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d30c61-40a8-4894-b554-1f4d75916150-maas-api-openshift-ca\") pod \"11d30c61-40a8-4894-b554-1f4d75916150\" (UID: \"11d30c61-40a8-4894-b554-1f4d75916150\") " Nov 03 20:25:45.970774 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:45.970680 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d30c61-40a8-4894-b554-1f4d75916150-kube-api-access-vz45p\" (UniqueName: \"kubernetes.io/projected/11d30c61-40a8-4894-b554-1f4d75916150-kube-api-access-vz45p\") pod \"11d30c61-40a8-4894-b554-1f4d75916150\" (UID: \"11d30c61-40a8-4894-b554-1f4d75916150\") " Nov 03 20:25:45.970979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:45.970949 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d30c61-40a8-4894-b554-1f4d75916150-maas-api-openshift-ca" pod "11d30c61-40a8-4894-b554-1f4d75916150" (UID: "11d30c61-40a8-4894-b554-1f4d75916150"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:45.971430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:45.971401 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d30c61-40a8-4894-b554-1f4d75916150-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:45.972978 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:45.972952 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d30c61-40a8-4894-b554-1f4d75916150-maas-api-xks-ca" pod "11d30c61-40a8-4894-b554-1f4d75916150" (UID: "11d30c61-40a8-4894-b554-1f4d75916150"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:45.973371 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:45.973347 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d30c61-40a8-4894-b554-1f4d75916150-kube-api-access-vz45p" pod "11d30c61-40a8-4894-b554-1f4d75916150" (UID: "11d30c61-40a8-4894-b554-1f4d75916150"). InnerVolumeSpecName "kube-api-access-vz45p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:46.064424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.064383 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb"] Nov 03 20:25:46.065673 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.065640 5111 cpu_manager.go:414] "containerMap: removing container" podUID="11d30c61-40a8-4894-b554-1f4d75916150" containerName="revoke-keys" Nov 03 20:25:46.065914 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.065884 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d30c61-40a8-4894-b554-1f4d75916150" containerName="revoke-keys" Nov 03 20:25:46.066062 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.066047 5111 cpu_manager.go:414] "containerMap: removing container" podUID="b7fc530f-7898-450f-bc05-be85b74215cf" containerName="istio-proxy" Nov 03 20:25:46.066106 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.066063 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7fc530f-7898-450f-bc05-be85b74215cf" containerName="istio-proxy" Nov 03 20:25:46.066106 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.066083 5111 cpu_manager.go:414] "containerMap: removing container" podUID="8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" containerName="istio-proxy" Nov 03 20:25:46.066106 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.066091 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" containerName="istio-proxy" Nov 03 20:25:46.066366 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.066339 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7fc530f-7898-450f-bc05-be85b74215cf" containerName="istio-proxy" Nov 03 20:25:46.066366 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.066360 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de95fa5-fdc8-4b2b-bbe9-a40b2d4672c7" containerName="istio-proxy" Nov 03 20:25:46.066527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.066372 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d30c61-40a8-4894-b554-1f4d75916150" containerName="revoke-keys" Nov 03 20:25:46.071212 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.071189 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb" Nov 03 20:25:46.072614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.072588 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4xsg\" (UniqueName: \"kubernetes.io/projected/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-kube-api-access-s4xsg\") pod \"maas-api-revoke-sub-a8a9eb5a-hnsgb\" (UID: \"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb" Nov 03 20:25:46.072995 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.072973 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8a9eb5a-hnsgb\" (UID: \"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb" Nov 03 20:25:46.073215 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.073188 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8a9eb5a-hnsgb\" (UID: \"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb" Nov 03 20:25:46.073426 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.073410 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d30c61-40a8-4894-b554-1f4d75916150-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:46.073674 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.073644 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz45p\" (UniqueName: \"kubernetes.io/projected/11d30c61-40a8-4894-b554-1f4d75916150-kube-api-access-vz45p\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:46.080488 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.079791 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb"] Nov 03 20:25:46.114112 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.112815 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" event={"ID":"11d30c61-40a8-4894-b554-1f4d75916150","Type":"ContainerDied","Data":"5f84d573fca40e39fca7ab098d9c3af33e183cc0a98a861bb2946386a6661f63"} Nov 03 20:25:46.114112 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.112909 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f84d573fca40e39fca7ab098d9c3af33e183cc0a98a861bb2946386a6661f63" Nov 03 20:25:46.114112 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.113035 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq" Nov 03 20:25:46.174902 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.174865 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8a9eb5a-hnsgb\" (UID: \"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb" Nov 03 20:25:46.175090 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.174974 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8a9eb5a-hnsgb\" (UID: \"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb" Nov 03 20:25:46.175150 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.175130 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4xsg\" (UniqueName: \"kubernetes.io/projected/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-kube-api-access-s4xsg\") pod \"maas-api-revoke-sub-a8a9eb5a-hnsgb\" (UID: \"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb" Nov 03 20:25:46.177073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.177045 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8a9eb5a-hnsgb\" (UID: \"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb" Nov 03 20:25:46.178946 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.178917 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8a9eb5a-hnsgb\" (UID: \"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb" Nov 03 20:25:46.184917 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.184895 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4xsg\" (UniqueName: \"kubernetes.io/projected/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-kube-api-access-s4xsg\") pod \"maas-api-revoke-sub-a8a9eb5a-hnsgb\" (UID: \"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb" Nov 03 20:25:46.188951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.188926 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca18671-d98f-405b-b297-6a0b09a217be" path="/var/lib/kubelet/pods/5ca18671-d98f-405b-b297-6a0b09a217be/volumes" Nov 03 20:25:46.408318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.408284 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb" Nov 03 20:25:46.724369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:46.724338 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb"] Nov 03 20:25:46.725281 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:25:46.725253 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52854a5d_5b2b_4b2f_85d5_889b7b0e6eb2.slice/crio-d4746832434f82cbfd2b986894ac8a4d556bd0edb08ab751a37dbcec69cff447 WatchSource:0}: Error finding container d4746832434f82cbfd2b986894ac8a4d556bd0edb08ab751a37dbcec69cff447: Status 404 returned error can't find the container with id d4746832434f82cbfd2b986894ac8a4d556bd0edb08ab751a37dbcec69cff447 Nov 03 20:25:47.125981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:47.125890 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb" event={"ID":"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2","Type":"ContainerStarted","Data":"d4746832434f82cbfd2b986894ac8a4d556bd0edb08ab751a37dbcec69cff447"} Nov 03 20:25:48.138241 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:48.138207 5111 generic.go:356] "Generic (PLEG): container finished" podID="52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2" containerID="6e984e87444e88b4bb9ad48fac3dc470f9bf319cf62b8b0e04fecca240f8a7f4" exitCode=0 Nov 03 20:25:48.138730 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:48.138292 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb" event={"ID":"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2","Type":"ContainerDied","Data":"6e984e87444e88b4bb9ad48fac3dc470f9bf319cf62b8b0e04fecca240f8a7f4"} Nov 03 20:25:49.340824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:49.340779 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd"] Nov 03 20:25:49.364408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:49.364367 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7343-mdbzd"] Nov 03 20:25:49.632764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:49.632730 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb" Nov 03 20:25:49.660900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:49.660867 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-maas-api-openshift-ca\") pod \"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2\" (UID: \"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2\") " Nov 03 20:25:49.661052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:49.660967 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-kube-api-access-s4xsg\" (UniqueName: \"kubernetes.io/projected/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-kube-api-access-s4xsg\") pod \"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2\" (UID: \"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2\") " Nov 03 20:25:49.661113 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:49.661072 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-maas-api-xks-ca\") pod \"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2\" (UID: \"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2\") " Nov 03 20:25:49.661327 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:49.661293 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-maas-api-openshift-ca" pod "52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2" (UID: "52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:49.661874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:49.661853 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:49.670590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:49.663527 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-maas-api-xks-ca" pod "52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2" (UID: "52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:49.670590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:49.663711 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-kube-api-access-s4xsg" pod "52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2" (UID: "52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2"). InnerVolumeSpecName "kube-api-access-s4xsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:49.763968 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:49.763930 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4xsg\" (UniqueName: \"kubernetes.io/projected/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-kube-api-access-s4xsg\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:49.763968 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:49.763968 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:50.188231 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:50.188197 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac5fa53-a2ee-4811-8eff-fbd745e00a7d" path="/var/lib/kubelet/pods/6ac5fa53-a2ee-4811-8eff-fbd745e00a7d/volumes" Nov 03 20:25:50.358713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:50.358681 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb" Nov 03 20:25:50.359139 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:50.358681 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb" event={"ID":"52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2","Type":"ContainerDied","Data":"d4746832434f82cbfd2b986894ac8a4d556bd0edb08ab751a37dbcec69cff447"} Nov 03 20:25:50.359139 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:50.358811 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4746832434f82cbfd2b986894ac8a4d556bd0edb08ab751a37dbcec69cff447" Nov 03 20:25:51.506378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:51.506318 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p"] Nov 03 20:25:51.522304 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:51.522255 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb"] Nov 03 20:25:52.253557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.252836 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57"] Nov 03 20:25:52.253557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.253084 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" podUID="7608cd01-af9c-4819-9016-6c010e0615ec" containerName="istio-proxy" containerID="cri-o://8025b778a3669dc573ea6a44cb67407245fc582d27f0148a58c14a7f7f174a11" gracePeriod=30 Nov 03 20:25:52.258392 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.257609 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx"] Nov 03 20:25:52.258392 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.257812 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" podUID="45038f2d-5a7b-4f31-92f3-147292e1c517" containerName="istio-proxy" containerID="cri-o://5fc029674e09a18493b9cfb5cda4b9322291acf45ce1dfcb5eb662eb7924a57c" gracePeriod=30 Nov 03 20:25:52.268266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.266927 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx"] Nov 03 20:25:52.268266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.267166 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" podUID="95b31fd8-6624-466d-a5cf-2c84b5305b29" containerName="istio-proxy" containerID="cri-o://87087700fc59440bbde04edafa88932f94d7e8716e085e723f818a99710bcb45" gracePeriod=30 Nov 03 20:25:52.276099 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.276060 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr"] Nov 03 20:25:52.276383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.276323 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" podUID="5d4d92cc-f4fa-421c-8b95-89838debd9b4" containerName="istio-proxy" containerID="cri-o://e47df2ee3f51faaea58722cc8fe986a244c67c82605ea24fa0cf50399fbcda5d" gracePeriod=30 Nov 03 20:25:52.741120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.741082 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh"] Nov 03 20:25:52.741958 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.741942 5111 cpu_manager.go:414] "containerMap: removing container" podUID="52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2" containerName="revoke-keys" Nov 03 20:25:52.742020 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.741959 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2" containerName="revoke-keys" Nov 03 20:25:52.742105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.742096 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2" containerName="revoke-keys" Nov 03 20:25:52.747505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.747476 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" Nov 03 20:25:52.751185 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.751162 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:25:52.751545 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.751510 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:25:52.757913 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.757883 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh"] Nov 03 20:25:52.846866 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.846830 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh\" (UID: \"0947ae5b-4856-4c5a-88dc-7ffc8321b06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" Nov 03 20:25:52.847091 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.847065 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh\" (UID: \"0947ae5b-4856-4c5a-88dc-7ffc8321b06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" Nov 03 20:25:52.847185 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.847114 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nrwl\" (UniqueName: \"kubernetes.io/projected/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-kube-api-access-5nrwl\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh\" (UID: \"0947ae5b-4856-4c5a-88dc-7ffc8321b06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" Nov 03 20:25:52.948239 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.948204 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh\" (UID: \"0947ae5b-4856-4c5a-88dc-7ffc8321b06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" Nov 03 20:25:52.948468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.948257 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nrwl\" (UniqueName: \"kubernetes.io/projected/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-kube-api-access-5nrwl\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh\" (UID: \"0947ae5b-4856-4c5a-88dc-7ffc8321b06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" Nov 03 20:25:52.948468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.948420 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh\" (UID: \"0947ae5b-4856-4c5a-88dc-7ffc8321b06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" Nov 03 20:25:52.949116 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.949092 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh\" (UID: \"0947ae5b-4856-4c5a-88dc-7ffc8321b06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" Nov 03 20:25:52.950858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.950829 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh\" (UID: \"0947ae5b-4856-4c5a-88dc-7ffc8321b06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" Nov 03 20:25:52.956345 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:52.956320 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nrwl\" (UniqueName: \"kubernetes.io/projected/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-kube-api-access-5nrwl\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh\" (UID: \"0947ae5b-4856-4c5a-88dc-7ffc8321b06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" Nov 03 20:25:53.075711 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:53.075623 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" Nov 03 20:25:53.409704 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:53.409669 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh"] Nov 03 20:25:53.410228 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:25:53.410201 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0947ae5b_4856_4c5a_88dc_7ffc8321b06f.slice/crio-b018dd1267a7df0b109c326e64849d3732075844e578105756eda1e1ffecffba WatchSource:0}: Error finding container b018dd1267a7df0b109c326e64849d3732075844e578105756eda1e1ffecffba: Status 404 returned error can't find the container with id b018dd1267a7df0b109c326e64849d3732075844e578105756eda1e1ffecffba Nov 03 20:25:53.576076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:53.576037 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" event={"ID":"0947ae5b-4856-4c5a-88dc-7ffc8321b06f","Type":"ContainerStarted","Data":"b018dd1267a7df0b109c326e64849d3732075844e578105756eda1e1ffecffba"} Nov 03 20:25:54.587306 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:54.587265 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" event={"ID":"0947ae5b-4856-4c5a-88dc-7ffc8321b06f","Type":"ContainerStarted","Data":"daa367ca3bf93e4fdbf3733d0c9b14c2d60e3dc3e109d1c731d1c87f077d8275"} Nov 03 20:25:54.607054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:54.606993 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" podStartSLOduration=2.606980151 podStartE2EDuration="2.606980151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:54.604221333 +0000 UTC m=+2784.566599770" watchObservedRunningTime="2026-11-03 20:25:54.606980151 +0000 UTC m=+2784.569358630" Nov 03 20:25:55.598811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:55.598774 5111 generic.go:356] "Generic (PLEG): container finished" podID="0947ae5b-4856-4c5a-88dc-7ffc8321b06f" containerID="daa367ca3bf93e4fdbf3733d0c9b14c2d60e3dc3e109d1c731d1c87f077d8275" exitCode=0 Nov 03 20:25:55.599437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:55.599145 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" event={"ID":"0947ae5b-4856-4c5a-88dc-7ffc8321b06f","Type":"ContainerDied","Data":"daa367ca3bf93e4fdbf3733d0c9b14c2d60e3dc3e109d1c731d1c87f077d8275"} Nov 03 20:25:56.361700 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:56.361660 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl"] Nov 03 20:25:56.361937 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:56.361903 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" podUID="6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" containerName="istio-proxy" containerID="cri-o://6ed9c0d6de305e562e5855fbee5080107934d155639e2801bde5743c7c659ef5" gracePeriod=30 Nov 03 20:25:56.508898 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:56.508852 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" podUID="2aa57a2e-9b53-4c9e-a17c-48f12faaac69" containerName="payload-processing" containerID="cri-o://f1d3f087ba08d53554d5d78716981a646d6b0bd78515a3fa1f404feef6c7190b" gracePeriod=25 Nov 03 20:25:56.529571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:56.527712 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" podUID="407c2001-b858-4378-a649-c300979e4f85" containerName="payload-pre-processing" containerID="cri-o://9e0b64c0a06dfefd53aa759f7b81202409a6ef646931d4072191e8fda51503c3" gracePeriod=25 Nov 03 20:25:56.867737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:56.867702 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb"] Nov 03 20:25:57.110592 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.107089 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb"] Nov 03 20:25:57.110592 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.107234 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb" Nov 03 20:25:57.152823 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.152770 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5"] Nov 03 20:25:57.160720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.160641 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" Nov 03 20:25:57.172226 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.172200 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5"] Nov 03 20:25:57.173398 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.173378 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5" Nov 03 20:25:57.201890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.200782 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" Nov 03 20:25:57.207878 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.207845 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" Nov 03 20:25:57.216147 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.216113 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlp6k\" (UniqueName: \"kubernetes.io/projected/afe71f33-2cb2-437e-88b0-f544c9cbe663-kube-api-access-rlp6k\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb\" (UID: \"afe71f33-2cb2-437e-88b0-f544c9cbe663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb" Nov 03 20:25:57.216824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.216798 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe71f33-2cb2-437e-88b0-f544c9cbe663-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb\" (UID: \"afe71f33-2cb2-437e-88b0-f544c9cbe663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb" Nov 03 20:25:57.217201 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.217185 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe71f33-2cb2-437e-88b0-f544c9cbe663-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb\" (UID: \"afe71f33-2cb2-437e-88b0-f544c9cbe663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb" Nov 03 20:25:57.318250 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.318210 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407c2001-b858-4378-a649-c300979e4f85-kube-api-access-bbdfc\" (UniqueName: \"kubernetes.io/projected/407c2001-b858-4378-a649-c300979e4f85-kube-api-access-bbdfc\") pod \"407c2001-b858-4378-a649-c300979e4f85\" (UID: \"407c2001-b858-4378-a649-c300979e4f85\") " Nov 03 20:25:57.318433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.318291 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa57a2e-9b53-4c9e-a17c-48f12faaac69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa57a2e-9b53-4c9e-a17c-48f12faaac69-plugins-config-volume\") pod \"2aa57a2e-9b53-4c9e-a17c-48f12faaac69\" (UID: \"2aa57a2e-9b53-4c9e-a17c-48f12faaac69\") " Nov 03 20:25:57.318433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.318381 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa57a2e-9b53-4c9e-a17c-48f12faaac69-kube-api-access-58qvs\" (UniqueName: \"kubernetes.io/projected/2aa57a2e-9b53-4c9e-a17c-48f12faaac69-kube-api-access-58qvs\") pod \"2aa57a2e-9b53-4c9e-a17c-48f12faaac69\" (UID: \"2aa57a2e-9b53-4c9e-a17c-48f12faaac69\") " Nov 03 20:25:57.318433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.318405 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-kube-api-access-5nrwl\" (UniqueName: \"kubernetes.io/projected/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-kube-api-access-5nrwl\") pod \"0947ae5b-4856-4c5a-88dc-7ffc8321b06f\" (UID: \"0947ae5b-4856-4c5a-88dc-7ffc8321b06f\") " Nov 03 20:25:57.318580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.318561 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-maas-api-xks-ca\") pod \"0947ae5b-4856-4c5a-88dc-7ffc8321b06f\" (UID: \"0947ae5b-4856-4c5a-88dc-7ffc8321b06f\") " Nov 03 20:25:57.318681 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.318664 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407c2001-b858-4378-a649-c300979e4f85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407c2001-b858-4378-a649-c300979e4f85-plugins-config-volume\") pod \"407c2001-b858-4378-a649-c300979e4f85\" (UID: \"407c2001-b858-4378-a649-c300979e4f85\") " Nov 03 20:25:57.318794 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.318758 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa57a2e-9b53-4c9e-a17c-48f12faaac69-plugins-config-volume" pod "2aa57a2e-9b53-4c9e-a17c-48f12faaac69" (UID: "2aa57a2e-9b53-4c9e-a17c-48f12faaac69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:57.318945 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.318838 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-maas-api-openshift-ca\") pod \"0947ae5b-4856-4c5a-88dc-7ffc8321b06f\" (UID: \"0947ae5b-4856-4c5a-88dc-7ffc8321b06f\") " Nov 03 20:25:57.319119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.319074 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407c2001-b858-4378-a649-c300979e4f85-plugins-config-volume" pod "407c2001-b858-4378-a649-c300979e4f85" (UID: "407c2001-b858-4378-a649-c300979e4f85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:57.319270 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.319247 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe71f33-2cb2-437e-88b0-f544c9cbe663-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb\" (UID: \"afe71f33-2cb2-437e-88b0-f544c9cbe663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb" Nov 03 20:25:57.319422 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.319322 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61778c4-0880-4606-a5af-0ef731921ead-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22a567-25bb5\" (UID: \"c61778c4-0880-4606-a5af-0ef731921ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5" Nov 03 20:25:57.319422 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.319326 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-maas-api-openshift-ca" pod "0947ae5b-4856-4c5a-88dc-7ffc8321b06f" (UID: "0947ae5b-4856-4c5a-88dc-7ffc8321b06f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:57.319681 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.319447 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-444wb\" (UniqueName: \"kubernetes.io/projected/c61778c4-0880-4606-a5af-0ef731921ead-kube-api-access-444wb\") pod \"maas-api-revoke-keys-e2e-inf-a-22a567-25bb5\" (UID: \"c61778c4-0880-4606-a5af-0ef731921ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5" Nov 03 20:25:57.319787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.319683 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlp6k\" (UniqueName: \"kubernetes.io/projected/afe71f33-2cb2-437e-88b0-f544c9cbe663-kube-api-access-rlp6k\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb\" (UID: \"afe71f33-2cb2-437e-88b0-f544c9cbe663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb" Nov 03 20:25:57.319892 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.319877 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe71f33-2cb2-437e-88b0-f544c9cbe663-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb\" (UID: \"afe71f33-2cb2-437e-88b0-f544c9cbe663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb" Nov 03 20:25:57.319955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.319938 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61778c4-0880-4606-a5af-0ef731921ead-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22a567-25bb5\" (UID: \"c61778c4-0880-4606-a5af-0ef731921ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5" Nov 03 20:25:57.320133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.320109 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe71f33-2cb2-437e-88b0-f544c9cbe663-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb\" (UID: \"afe71f33-2cb2-437e-88b0-f544c9cbe663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb" Nov 03 20:25:57.320133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.320126 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa57a2e-9b53-4c9e-a17c-48f12faaac69-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:57.320257 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.320222 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407c2001-b858-4378-a649-c300979e4f85-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:57.320257 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.320244 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:57.321339 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.321314 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407c2001-b858-4378-a649-c300979e4f85-kube-api-access-bbdfc" pod "407c2001-b858-4378-a649-c300979e4f85" (UID: "407c2001-b858-4378-a649-c300979e4f85"). InnerVolumeSpecName "kube-api-access-bbdfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:57.321434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.321355 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa57a2e-9b53-4c9e-a17c-48f12faaac69-kube-api-access-58qvs" pod "2aa57a2e-9b53-4c9e-a17c-48f12faaac69" (UID: "2aa57a2e-9b53-4c9e-a17c-48f12faaac69"). InnerVolumeSpecName "kube-api-access-58qvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:57.321479 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.321441 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-kube-api-access-5nrwl" pod "0947ae5b-4856-4c5a-88dc-7ffc8321b06f" (UID: "0947ae5b-4856-4c5a-88dc-7ffc8321b06f"). InnerVolumeSpecName "kube-api-access-5nrwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:57.321584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.321518 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-maas-api-xks-ca" pod "0947ae5b-4856-4c5a-88dc-7ffc8321b06f" (UID: "0947ae5b-4856-4c5a-88dc-7ffc8321b06f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:57.322152 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.322129 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe71f33-2cb2-437e-88b0-f544c9cbe663-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb\" (UID: \"afe71f33-2cb2-437e-88b0-f544c9cbe663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb" Nov 03 20:25:57.327030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.326997 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlp6k\" (UniqueName: \"kubernetes.io/projected/afe71f33-2cb2-437e-88b0-f544c9cbe663-kube-api-access-rlp6k\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb\" (UID: \"afe71f33-2cb2-437e-88b0-f544c9cbe663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb" Nov 03 20:25:57.425787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.424783 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61778c4-0880-4606-a5af-0ef731921ead-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22a567-25bb5\" (UID: \"c61778c4-0880-4606-a5af-0ef731921ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5" Nov 03 20:25:57.427296 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.426315 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-444wb\" (UniqueName: \"kubernetes.io/projected/c61778c4-0880-4606-a5af-0ef731921ead-kube-api-access-444wb\") pod \"maas-api-revoke-keys-e2e-inf-a-22a567-25bb5\" (UID: \"c61778c4-0880-4606-a5af-0ef731921ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5" Nov 03 20:25:57.429287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.427717 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61778c4-0880-4606-a5af-0ef731921ead-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22a567-25bb5\" (UID: \"c61778c4-0880-4606-a5af-0ef731921ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5" Nov 03 20:25:57.429287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.427952 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbdfc\" (UniqueName: \"kubernetes.io/projected/407c2001-b858-4378-a649-c300979e4f85-kube-api-access-bbdfc\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:57.429287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.427967 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58qvs\" (UniqueName: \"kubernetes.io/projected/2aa57a2e-9b53-4c9e-a17c-48f12faaac69-kube-api-access-58qvs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:57.429287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.427978 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nrwl\" (UniqueName: \"kubernetes.io/projected/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-kube-api-access-5nrwl\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:57.429287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.427990 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0947ae5b-4856-4c5a-88dc-7ffc8321b06f-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:57.429287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.428583 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61778c4-0880-4606-a5af-0ef731921ead-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22a567-25bb5\" (UID: \"c61778c4-0880-4606-a5af-0ef731921ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5" Nov 03 20:25:57.438986 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.438948 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-444wb\" (UniqueName: \"kubernetes.io/projected/c61778c4-0880-4606-a5af-0ef731921ead-kube-api-access-444wb\") pod \"maas-api-revoke-keys-e2e-inf-a-22a567-25bb5\" (UID: \"c61778c4-0880-4606-a5af-0ef731921ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5" Nov 03 20:25:57.439760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.439732 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61778c4-0880-4606-a5af-0ef731921ead-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22a567-25bb5\" (UID: \"c61778c4-0880-4606-a5af-0ef731921ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5" Nov 03 20:25:57.460178 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.460143 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk"] Nov 03 20:25:57.460623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.460332 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" podUID="407008ca-4d98-4538-9f2c-bfdd9b03535a" containerName="discovery" containerID="cri-o://0c52f1466eeed8d7780f351a8661fd0c7b95439f189a7b03a7353411f5836ccf" gracePeriod=30 Nov 03 20:25:57.488789 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.486070 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb" Nov 03 20:25:57.533601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.533577 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5" Nov 03 20:25:57.697402 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.697367 5111 generic.go:356] "Generic (PLEG): container finished" podID="45038f2d-5a7b-4f31-92f3-147292e1c517" containerID="5fc029674e09a18493b9cfb5cda4b9322291acf45ce1dfcb5eb662eb7924a57c" exitCode=0 Nov 03 20:25:57.697579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.697514 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" event={"ID":"45038f2d-5a7b-4f31-92f3-147292e1c517","Type":"ContainerDied","Data":"5fc029674e09a18493b9cfb5cda4b9322291acf45ce1dfcb5eb662eb7924a57c"} Nov 03 20:25:57.699701 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.699513 5111 generic.go:356] "Generic (PLEG): container finished" podID="7608cd01-af9c-4819-9016-6c010e0615ec" containerID="8025b778a3669dc573ea6a44cb67407245fc582d27f0148a58c14a7f7f174a11" exitCode=0 Nov 03 20:25:57.699701 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.699632 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" event={"ID":"7608cd01-af9c-4819-9016-6c010e0615ec","Type":"ContainerDied","Data":"8025b778a3669dc573ea6a44cb67407245fc582d27f0148a58c14a7f7f174a11"} Nov 03 20:25:57.702588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.702520 5111 generic.go:356] "Generic (PLEG): container finished" podID="407008ca-4d98-4538-9f2c-bfdd9b03535a" containerID="0c52f1466eeed8d7780f351a8661fd0c7b95439f189a7b03a7353411f5836ccf" exitCode=0 Nov 03 20:25:57.702724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.702698 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" event={"ID":"407008ca-4d98-4538-9f2c-bfdd9b03535a","Type":"ContainerDied","Data":"0c52f1466eeed8d7780f351a8661fd0c7b95439f189a7b03a7353411f5836ccf"} Nov 03 20:25:57.704188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.704170 5111 generic.go:356] "Generic (PLEG): container finished" podID="407c2001-b858-4378-a649-c300979e4f85" containerID="9e0b64c0a06dfefd53aa759f7b81202409a6ef646931d4072191e8fda51503c3" exitCode=0 Nov 03 20:25:57.704411 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.704393 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" event={"ID":"407c2001-b858-4378-a649-c300979e4f85","Type":"ContainerDied","Data":"9e0b64c0a06dfefd53aa759f7b81202409a6ef646931d4072191e8fda51503c3"} Nov 03 20:25:57.704570 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.704554 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" event={"ID":"407c2001-b858-4378-a649-c300979e4f85","Type":"ContainerDied","Data":"ffa2fa11a0537401821e84ea7be80be38622be4ae4fc1bbcb3a3681784856795"} Nov 03 20:25:57.704822 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.704805 5111 scope.go:122] "RemoveContainer" containerID="9e0b64c0a06dfefd53aa759f7b81202409a6ef646931d4072191e8fda51503c3" Nov 03 20:25:57.705182 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.705166 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb" Nov 03 20:25:57.730024 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.719847 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" event={"ID":"0947ae5b-4856-4c5a-88dc-7ffc8321b06f","Type":"ContainerDied","Data":"b018dd1267a7df0b109c326e64849d3732075844e578105756eda1e1ffecffba"} Nov 03 20:25:57.730024 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.719921 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b018dd1267a7df0b109c326e64849d3732075844e578105756eda1e1ffecffba" Nov 03 20:25:57.730024 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.720005 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh" Nov 03 20:25:57.735633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.734598 5111 generic.go:356] "Generic (PLEG): container finished" podID="5d4d92cc-f4fa-421c-8b95-89838debd9b4" containerID="e47df2ee3f51faaea58722cc8fe986a244c67c82605ea24fa0cf50399fbcda5d" exitCode=0 Nov 03 20:25:57.735633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.734723 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" event={"ID":"5d4d92cc-f4fa-421c-8b95-89838debd9b4","Type":"ContainerDied","Data":"e47df2ee3f51faaea58722cc8fe986a244c67c82605ea24fa0cf50399fbcda5d"} Nov 03 20:25:57.738041 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.738019 5111 generic.go:356] "Generic (PLEG): container finished" podID="2aa57a2e-9b53-4c9e-a17c-48f12faaac69" containerID="f1d3f087ba08d53554d5d78716981a646d6b0bd78515a3fa1f404feef6c7190b" exitCode=0 Nov 03 20:25:57.738185 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.738169 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" Nov 03 20:25:57.741193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.741171 5111 generic.go:356] "Generic (PLEG): container finished" podID="95b31fd8-6624-466d-a5cf-2c84b5305b29" containerID="87087700fc59440bbde04edafa88932f94d7e8716e085e723f818a99710bcb45" exitCode=0 Nov 03 20:25:57.741613 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.741579 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" event={"ID":"2aa57a2e-9b53-4c9e-a17c-48f12faaac69","Type":"ContainerDied","Data":"f1d3f087ba08d53554d5d78716981a646d6b0bd78515a3fa1f404feef6c7190b"} Nov 03 20:25:57.741714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.741650 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p" event={"ID":"2aa57a2e-9b53-4c9e-a17c-48f12faaac69","Type":"ContainerDied","Data":"e94428cb0e3b09918bf4c728c505228c57ab84a6da3b4ace11b481eddce48090"} Nov 03 20:25:57.741714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.741678 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" event={"ID":"95b31fd8-6624-466d-a5cf-2c84b5305b29","Type":"ContainerDied","Data":"87087700fc59440bbde04edafa88932f94d7e8716e085e723f818a99710bcb45"} Nov 03 20:25:57.821920 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.799909 5111 scope.go:122] "RemoveContainer" containerID="9e0b64c0a06dfefd53aa759f7b81202409a6ef646931d4072191e8fda51503c3" Nov 03 20:25:57.920928 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:25:57.864125 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0b64c0a06dfefd53aa759f7b81202409a6ef646931d4072191e8fda51503c3\": container with ID starting with 9e0b64c0a06dfefd53aa759f7b81202409a6ef646931d4072191e8fda51503c3 not found: ID does not exist" containerID="9e0b64c0a06dfefd53aa759f7b81202409a6ef646931d4072191e8fda51503c3" Nov 03 20:25:57.920928 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.864177 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0b64c0a06dfefd53aa759f7b81202409a6ef646931d4072191e8fda51503c3"} err="failed to get container status \"9e0b64c0a06dfefd53aa759f7b81202409a6ef646931d4072191e8fda51503c3\": rpc error: code = NotFound desc = could not find container \"9e0b64c0a06dfefd53aa759f7b81202409a6ef646931d4072191e8fda51503c3\": container with ID starting with 9e0b64c0a06dfefd53aa759f7b81202409a6ef646931d4072191e8fda51503c3 not found: ID does not exist" Nov 03 20:25:57.920928 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.864203 5111 scope.go:122] "RemoveContainer" containerID="f1d3f087ba08d53554d5d78716981a646d6b0bd78515a3fa1f404feef6c7190b" Nov 03 20:25:57.920928 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.901749 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:25:57.920928 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.913692 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p"] Nov 03 20:25:57.959589 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.954476 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cad531-5c6bf4fb9d-wzt8p"] Nov 03 20:25:57.990829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.990007 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-workload-socket\") pod \"45038f2d-5a7b-4f31-92f3-147292e1c517\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " Nov 03 20:25:57.990829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.990131 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-token\" (UniqueName: \"kubernetes.io/projected/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-token\") pod \"45038f2d-5a7b-4f31-92f3-147292e1c517\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " Nov 03 20:25:57.990829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.990179 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-workload-certs\") pod \"45038f2d-5a7b-4f31-92f3-147292e1c517\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " Nov 03 20:25:57.990829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.990308 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-envoy\") pod \"45038f2d-5a7b-4f31-92f3-147292e1c517\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " Nov 03 20:25:57.990829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.990484 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45038f2d-5a7b-4f31-92f3-147292e1c517-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45038f2d-5a7b-4f31-92f3-147292e1c517-istiod-ca-cert\") pod \"45038f2d-5a7b-4f31-92f3-147292e1c517\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " Nov 03 20:25:57.990829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.990584 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-credential-socket\") pod \"45038f2d-5a7b-4f31-92f3-147292e1c517\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " Nov 03 20:25:57.992611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.991127 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-podinfo\") pod \"45038f2d-5a7b-4f31-92f3-147292e1c517\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " Nov 03 20:25:57.992611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.991256 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45038f2d-5a7b-4f31-92f3-147292e1c517-kube-api-access-cqxzd\" (UniqueName: \"kubernetes.io/projected/45038f2d-5a7b-4f31-92f3-147292e1c517-kube-api-access-cqxzd\") pod \"45038f2d-5a7b-4f31-92f3-147292e1c517\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " Nov 03 20:25:57.992611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.991306 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-data\") pod \"45038f2d-5a7b-4f31-92f3-147292e1c517\" (UID: \"45038f2d-5a7b-4f31-92f3-147292e1c517\") " Nov 03 20:25:57.993699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.992813 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-data" pod "45038f2d-5a7b-4f31-92f3-147292e1c517" (UID: "45038f2d-5a7b-4f31-92f3-147292e1c517"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:57.994502 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:57.993874 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-workload-socket" pod "45038f2d-5a7b-4f31-92f3-147292e1c517" (UID: "45038f2d-5a7b-4f31-92f3-147292e1c517"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.012098 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.007852 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb"] Nov 03 20:25:58.022690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.011466 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-credential-socket" pod "45038f2d-5a7b-4f31-92f3-147292e1c517" (UID: "45038f2d-5a7b-4f31-92f3-147292e1c517"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.023000 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.015312 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45038f2d-5a7b-4f31-92f3-147292e1c517-istiod-ca-cert" pod "45038f2d-5a7b-4f31-92f3-147292e1c517" (UID: "45038f2d-5a7b-4f31-92f3-147292e1c517"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:58.023172 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.015468 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-workload-certs" pod "45038f2d-5a7b-4f31-92f3-147292e1c517" (UID: "45038f2d-5a7b-4f31-92f3-147292e1c517"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.023297 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.018518 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-podinfo" pod "45038f2d-5a7b-4f31-92f3-147292e1c517" (UID: "45038f2d-5a7b-4f31-92f3-147292e1c517"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:25:58.023397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.020774 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45038f2d-5a7b-4f31-92f3-147292e1c517-kube-api-access-cqxzd" pod "45038f2d-5a7b-4f31-92f3-147292e1c517" (UID: "45038f2d-5a7b-4f31-92f3-147292e1c517"). InnerVolumeSpecName "kube-api-access-cqxzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:58.071815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.071088 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-token" pod "45038f2d-5a7b-4f31-92f3-147292e1c517" (UID: "45038f2d-5a7b-4f31-92f3-147292e1c517"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:58.071815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.071745 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cad531-5758dcd487-mw5jb"] Nov 03 20:25:58.100067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.081723 5111 scope.go:122] "RemoveContainer" containerID="f1d3f087ba08d53554d5d78716981a646d6b0bd78515a3fa1f404feef6c7190b" Nov 03 20:25:58.100067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.098197 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-envoy" pod "45038f2d-5a7b-4f31-92f3-147292e1c517" (UID: "45038f2d-5a7b-4f31-92f3-147292e1c517"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.102067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.100709 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.102067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.100742 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45038f2d-5a7b-4f31-92f3-147292e1c517-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.102067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.100757 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.102067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.100771 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.102067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.100786 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqxzd\" (UniqueName: \"kubernetes.io/projected/45038f2d-5a7b-4f31-92f3-147292e1c517-kube-api-access-cqxzd\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.102067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.100799 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.102067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.100813 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.102067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.100827 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45038f2d-5a7b-4f31-92f3-147292e1c517-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.102067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.100841 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45038f2d-5a7b-4f31-92f3-147292e1c517-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.103200 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:25:58.103150 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d3f087ba08d53554d5d78716981a646d6b0bd78515a3fa1f404feef6c7190b\": container with ID starting with f1d3f087ba08d53554d5d78716981a646d6b0bd78515a3fa1f404feef6c7190b not found: ID does not exist" containerID="f1d3f087ba08d53554d5d78716981a646d6b0bd78515a3fa1f404feef6c7190b" Nov 03 20:25:58.103373 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.103344 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d3f087ba08d53554d5d78716981a646d6b0bd78515a3fa1f404feef6c7190b"} err="failed to get container status \"f1d3f087ba08d53554d5d78716981a646d6b0bd78515a3fa1f404feef6c7190b\": rpc error: code = NotFound desc = could not find container \"f1d3f087ba08d53554d5d78716981a646d6b0bd78515a3fa1f404feef6c7190b\": container with ID starting with f1d3f087ba08d53554d5d78716981a646d6b0bd78515a3fa1f404feef6c7190b not found: ID does not exist" Nov 03 20:25:58.128896 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.128862 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:25:58.131074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.131049 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:25:58.151692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.149698 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:25:58.151692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.149788 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq"] Nov 03 20:25:58.158909 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.158874 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cad531-rrmgq"] Nov 03 20:25:58.202074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.201511 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d30c61-40a8-4894-b554-1f4d75916150" path="/var/lib/kubelet/pods/11d30c61-40a8-4894-b554-1f4d75916150/volumes" Nov 03 20:25:58.202074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.201635 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-data\") pod \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " Nov 03 20:25:58.202074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.201717 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-workload-socket\") pod \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " Nov 03 20:25:58.202074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.201770 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-istio-data\") pod \"7608cd01-af9c-4819-9016-6c010e0615ec\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " Nov 03 20:25:58.202074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.201808 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7608cd01-af9c-4819-9016-6c010e0615ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7608cd01-af9c-4819-9016-6c010e0615ec-istio-podinfo\") pod \"7608cd01-af9c-4819-9016-6c010e0615ec\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " Nov 03 20:25:58.202074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.201934 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-podinfo\") pod \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " Nov 03 20:25:58.202074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.202003 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istiod-ca-cert\") pod \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " Nov 03 20:25:58.202074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.202039 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-data" pod "5d4d92cc-f4fa-421c-8b95-89838debd9b4" (UID: "5d4d92cc-f4fa-421c-8b95-89838debd9b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.202724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.202052 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-credential-socket\") pod \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " Nov 03 20:25:58.202724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.202470 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7608cd01-af9c-4819-9016-6c010e0615ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7608cd01-af9c-4819-9016-6c010e0615ec-istiod-ca-cert\") pod \"7608cd01-af9c-4819-9016-6c010e0615ec\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " Nov 03 20:25:58.202724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.202191 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-istio-data" pod "7608cd01-af9c-4819-9016-6c010e0615ec" (UID: "7608cd01-af9c-4819-9016-6c010e0615ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.202724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.202283 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-credential-socket" pod "5d4d92cc-f4fa-421c-8b95-89838debd9b4" (UID: "5d4d92cc-f4fa-421c-8b95-89838debd9b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.202724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.202624 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istiod-ca-cert" pod "5d4d92cc-f4fa-421c-8b95-89838debd9b4" (UID: "5d4d92cc-f4fa-421c-8b95-89838debd9b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:58.202724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.202662 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-workload-certs\") pod \"7608cd01-af9c-4819-9016-6c010e0615ec\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " Nov 03 20:25:58.203046 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.202739 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7608cd01-af9c-4819-9016-6c010e0615ec-istio-token\" (UniqueName: \"kubernetes.io/projected/7608cd01-af9c-4819-9016-6c010e0615ec-istio-token\") pod \"7608cd01-af9c-4819-9016-6c010e0615ec\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " Nov 03 20:25:58.203046 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.202828 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-istio-envoy\") pod \"7608cd01-af9c-4819-9016-6c010e0615ec\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " Nov 03 20:25:58.203046 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.202908 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa57a2e-9b53-4c9e-a17c-48f12faaac69" path="/var/lib/kubelet/pods/2aa57a2e-9b53-4c9e-a17c-48f12faaac69/volumes" Nov 03 20:25:58.203046 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.202919 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-workload-socket\") pod \"7608cd01-af9c-4819-9016-6c010e0615ec\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " Nov 03 20:25:58.203046 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.202963 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d4d92cc-f4fa-421c-8b95-89838debd9b4-kube-api-access-mcm5q\" (UniqueName: \"kubernetes.io/projected/5d4d92cc-f4fa-421c-8b95-89838debd9b4-kube-api-access-mcm5q\") pod \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " Nov 03 20:25:58.203046 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.203013 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-envoy\") pod \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " Nov 03 20:25:58.203046 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.203030 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-workload-certs" pod "7608cd01-af9c-4819-9016-6c010e0615ec" (UID: "7608cd01-af9c-4819-9016-6c010e0615ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.203358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.203051 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7608cd01-af9c-4819-9016-6c010e0615ec-kube-api-access-8qmfd\" (UniqueName: \"kubernetes.io/projected/7608cd01-af9c-4819-9016-6c010e0615ec-kube-api-access-8qmfd\") pod \"7608cd01-af9c-4819-9016-6c010e0615ec\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " Nov 03 20:25:58.203358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.203065 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-workload-socket" pod "5d4d92cc-f4fa-421c-8b95-89838debd9b4" (UID: "5d4d92cc-f4fa-421c-8b95-89838debd9b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.203358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.203131 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-token\" (UniqueName: \"kubernetes.io/projected/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-token\") pod \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " Nov 03 20:25:58.203358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.203196 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-workload-certs\") pod \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\" (UID: \"5d4d92cc-f4fa-421c-8b95-89838debd9b4\") " Nov 03 20:25:58.203358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.203230 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-credential-socket\") pod \"7608cd01-af9c-4819-9016-6c010e0615ec\" (UID: \"7608cd01-af9c-4819-9016-6c010e0615ec\") " Nov 03 20:25:58.203358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.203345 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-workload-socket" pod "7608cd01-af9c-4819-9016-6c010e0615ec" (UID: "7608cd01-af9c-4819-9016-6c010e0615ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.205002 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.203657 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-credential-socket" pod "7608cd01-af9c-4819-9016-6c010e0615ec" (UID: "7608cd01-af9c-4819-9016-6c010e0615ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.205002 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.203724 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7608cd01-af9c-4819-9016-6c010e0615ec-istiod-ca-cert" pod "7608cd01-af9c-4819-9016-6c010e0615ec" (UID: "7608cd01-af9c-4819-9016-6c010e0615ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:58.205002 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.203793 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407c2001-b858-4378-a649-c300979e4f85" path="/var/lib/kubelet/pods/407c2001-b858-4378-a649-c300979e4f85/volumes" Nov 03 20:25:58.205608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.205584 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.205688 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.205614 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7608cd01-af9c-4819-9016-6c010e0615ec-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.205688 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.205629 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.205688 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.205644 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.205688 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.205660 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.205688 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.205674 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.205688 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.205687 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.206037 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.205701 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.206037 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.205716 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.207426 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.207401 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-token" pod "5d4d92cc-f4fa-421c-8b95-89838debd9b4" (UID: "5d4d92cc-f4fa-421c-8b95-89838debd9b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:58.207661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.207637 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-workload-certs" pod "5d4d92cc-f4fa-421c-8b95-89838debd9b4" (UID: "5d4d92cc-f4fa-421c-8b95-89838debd9b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.208229 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.208206 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4d92cc-f4fa-421c-8b95-89838debd9b4-kube-api-access-mcm5q" pod "5d4d92cc-f4fa-421c-8b95-89838debd9b4" (UID: "5d4d92cc-f4fa-421c-8b95-89838debd9b4"). InnerVolumeSpecName "kube-api-access-mcm5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:58.208365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.208265 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7608cd01-af9c-4819-9016-6c010e0615ec-kube-api-access-8qmfd" pod "7608cd01-af9c-4819-9016-6c010e0615ec" (UID: "7608cd01-af9c-4819-9016-6c010e0615ec"). InnerVolumeSpecName "kube-api-access-8qmfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:58.215652 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.215618 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7608cd01-af9c-4819-9016-6c010e0615ec-istio-podinfo" pod "7608cd01-af9c-4819-9016-6c010e0615ec" (UID: "7608cd01-af9c-4819-9016-6c010e0615ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:25:58.215961 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.215940 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-podinfo" pod "5d4d92cc-f4fa-421c-8b95-89838debd9b4" (UID: "5d4d92cc-f4fa-421c-8b95-89838debd9b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:25:58.216219 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.216005 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-istio-envoy" pod "7608cd01-af9c-4819-9016-6c010e0615ec" (UID: "7608cd01-af9c-4819-9016-6c010e0615ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.216509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.216125 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-envoy" pod "5d4d92cc-f4fa-421c-8b95-89838debd9b4" (UID: "5d4d92cc-f4fa-421c-8b95-89838debd9b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.216640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.216161 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7608cd01-af9c-4819-9016-6c010e0615ec-istio-token" pod "7608cd01-af9c-4819-9016-6c010e0615ec" (UID: "7608cd01-af9c-4819-9016-6c010e0615ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:58.309130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.309097 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b31fd8-6624-466d-a5cf-2c84b5305b29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b31fd8-6624-466d-a5cf-2c84b5305b29-istiod-ca-cert\") pod \"95b31fd8-6624-466d-a5cf-2c84b5305b29\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " Nov 03 20:25:58.309316 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.309184 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-workload-certs\") pod \"95b31fd8-6624-466d-a5cf-2c84b5305b29\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " Nov 03 20:25:58.309316 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.309244 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-podinfo\") pod \"95b31fd8-6624-466d-a5cf-2c84b5305b29\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " Nov 03 20:25:58.309461 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.309338 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-workload-socket\") pod \"95b31fd8-6624-466d-a5cf-2c84b5305b29\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " Nov 03 20:25:58.309461 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.309374 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-token\" (UniqueName: \"kubernetes.io/projected/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-token\") pod \"95b31fd8-6624-466d-a5cf-2c84b5305b29\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " Nov 03 20:25:58.309461 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.309420 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b31fd8-6624-466d-a5cf-2c84b5305b29-kube-api-access-kxlz5\" (UniqueName: \"kubernetes.io/projected/95b31fd8-6624-466d-a5cf-2c84b5305b29-kube-api-access-kxlz5\") pod \"95b31fd8-6624-466d-a5cf-2c84b5305b29\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " Nov 03 20:25:58.309661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.309491 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-data\") pod \"95b31fd8-6624-466d-a5cf-2c84b5305b29\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " Nov 03 20:25:58.309661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.309649 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-credential-socket\") pod \"95b31fd8-6624-466d-a5cf-2c84b5305b29\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " Nov 03 20:25:58.309805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.309763 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-envoy\") pod \"95b31fd8-6624-466d-a5cf-2c84b5305b29\" (UID: \"95b31fd8-6624-466d-a5cf-2c84b5305b29\") " Nov 03 20:25:58.310560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.310526 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7608cd01-af9c-4819-9016-6c010e0615ec-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.310560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.310563 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7608cd01-af9c-4819-9016-6c010e0615ec-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.310560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.310588 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcm5q\" (UniqueName: \"kubernetes.io/projected/5d4d92cc-f4fa-421c-8b95-89838debd9b4-kube-api-access-mcm5q\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.310560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.310601 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.310916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.310633 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qmfd\" (UniqueName: \"kubernetes.io/projected/7608cd01-af9c-4819-9016-6c010e0615ec-kube-api-access-8qmfd\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.310916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.310646 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.310916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.310657 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d4d92cc-f4fa-421c-8b95-89838debd9b4-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.310916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.310671 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7608cd01-af9c-4819-9016-6c010e0615ec-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.310916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.310685 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d4d92cc-f4fa-421c-8b95-89838debd9b4-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.311243 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.311221 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b31fd8-6624-466d-a5cf-2c84b5305b29-istiod-ca-cert" pod "95b31fd8-6624-466d-a5cf-2c84b5305b29" (UID: "95b31fd8-6624-466d-a5cf-2c84b5305b29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:58.311480 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.311462 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-workload-certs" pod "95b31fd8-6624-466d-a5cf-2c84b5305b29" (UID: "95b31fd8-6624-466d-a5cf-2c84b5305b29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.313098 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.312849 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-workload-socket" pod "95b31fd8-6624-466d-a5cf-2c84b5305b29" (UID: "95b31fd8-6624-466d-a5cf-2c84b5305b29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.313098 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.313057 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-credential-socket" pod "95b31fd8-6624-466d-a5cf-2c84b5305b29" (UID: "95b31fd8-6624-466d-a5cf-2c84b5305b29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.313483 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.313462 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-data" pod "95b31fd8-6624-466d-a5cf-2c84b5305b29" (UID: "95b31fd8-6624-466d-a5cf-2c84b5305b29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.324474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.323699 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:25:58.324746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.324718 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-token" pod "95b31fd8-6624-466d-a5cf-2c84b5305b29" (UID: "95b31fd8-6624-466d-a5cf-2c84b5305b29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:58.333800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.333769 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-podinfo" pod "95b31fd8-6624-466d-a5cf-2c84b5305b29" (UID: "95b31fd8-6624-466d-a5cf-2c84b5305b29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:25:58.333800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.333768 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-envoy" pod "95b31fd8-6624-466d-a5cf-2c84b5305b29" (UID: "95b31fd8-6624-466d-a5cf-2c84b5305b29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.336911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.336863 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b31fd8-6624-466d-a5cf-2c84b5305b29-kube-api-access-kxlz5" pod "95b31fd8-6624-466d-a5cf-2c84b5305b29" (UID: "95b31fd8-6624-466d-a5cf-2c84b5305b29"). InnerVolumeSpecName "kube-api-access-kxlz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:58.411999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.411969 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-kubeconfig\") pod \"407008ca-4d98-4538-9f2c-bfdd9b03535a\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " Nov 03 20:25:58.412182 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.412105 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-cacerts\" (UniqueName: \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-cacerts\") pod \"407008ca-4d98-4538-9f2c-bfdd9b03535a\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " Nov 03 20:25:58.412255 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.412177 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407008ca-4d98-4538-9f2c-bfdd9b03535a-kube-api-access-9562k\" (UniqueName: \"kubernetes.io/projected/407008ca-4d98-4538-9f2c-bfdd9b03535a-kube-api-access-9562k\") pod \"407008ca-4d98-4538-9f2c-bfdd9b03535a\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " Nov 03 20:25:58.412255 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.412211 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-token\" (UniqueName: \"kubernetes.io/projected/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-token\") pod \"407008ca-4d98-4538-9f2c-bfdd9b03535a\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " Nov 03 20:25:58.412255 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.412241 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-csr-ca-configmap\") pod \"407008ca-4d98-4538-9f2c-bfdd9b03535a\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " Nov 03 20:25:58.412409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.412299 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407008ca-4d98-4538-9f2c-bfdd9b03535a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/407008ca-4d98-4538-9f2c-bfdd9b03535a-local-certs\") pod \"407008ca-4d98-4538-9f2c-bfdd9b03535a\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " Nov 03 20:25:58.412545 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.412507 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-csr-dns-cert\") pod \"407008ca-4d98-4538-9f2c-bfdd9b03535a\" (UID: \"407008ca-4d98-4538-9f2c-bfdd9b03535a\") " Nov 03 20:25:58.412789 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.412740 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-csr-ca-configmap" pod "407008ca-4d98-4538-9f2c-bfdd9b03535a" (UID: "407008ca-4d98-4538-9f2c-bfdd9b03535a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:58.413458 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.413429 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b31fd8-6624-466d-a5cf-2c84b5305b29-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.413671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.413462 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.413671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.413477 5111 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-csr-ca-configmap\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.413671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.413492 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.413671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.413506 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.413671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.413522 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.413671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.413577 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxlz5\" (UniqueName: \"kubernetes.io/projected/95b31fd8-6624-466d-a5cf-2c84b5305b29-kube-api-access-kxlz5\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.413671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.413595 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.413671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.413607 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.413671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.413622 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b31fd8-6624-466d-a5cf-2c84b5305b29-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.414904 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.414827 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407008ca-4d98-4538-9f2c-bfdd9b03535a-kube-api-access-9562k" pod "407008ca-4d98-4538-9f2c-bfdd9b03535a" (UID: "407008ca-4d98-4538-9f2c-bfdd9b03535a"). InnerVolumeSpecName "kube-api-access-9562k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:58.415030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.414965 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-token" pod "407008ca-4d98-4538-9f2c-bfdd9b03535a" (UID: "407008ca-4d98-4538-9f2c-bfdd9b03535a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:58.415030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.414964 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-kubeconfig" pod "407008ca-4d98-4538-9f2c-bfdd9b03535a" (UID: "407008ca-4d98-4538-9f2c-bfdd9b03535a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:58.416686 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.416662 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-csr-dns-cert" pod "407008ca-4d98-4538-9f2c-bfdd9b03535a" (UID: "407008ca-4d98-4538-9f2c-bfdd9b03535a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:58.416961 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.416939 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-cacerts" pod "407008ca-4d98-4538-9f2c-bfdd9b03535a" (UID: "407008ca-4d98-4538-9f2c-bfdd9b03535a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:58.417227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.417209 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407008ca-4d98-4538-9f2c-bfdd9b03535a-local-certs" pod "407008ca-4d98-4538-9f2c-bfdd9b03535a" (UID: "407008ca-4d98-4538-9f2c-bfdd9b03535a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:58.452495 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.452436 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb"] Nov 03 20:25:58.452714 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:25:58.452685 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe71f33_2cb2_437e_88b0_f544c9cbe663.slice/crio-d9df8bb1934327afd65e9e157428a54c87a13ac38955ceb687c55b66a468d6fc WatchSource:0}: Error finding container d9df8bb1934327afd65e9e157428a54c87a13ac38955ceb687c55b66a468d6fc: Status 404 returned error can't find the container with id d9df8bb1934327afd65e9e157428a54c87a13ac38955ceb687c55b66a468d6fc Nov 03 20:25:58.512196 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:25:58.512161 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61778c4_0880_4606_a5af_0ef731921ead.slice/crio-65f4cd43b72009bb8bbe535a0c994803dff988d7a454eca0abf5af4e3a3bb747 WatchSource:0}: Error finding container 65f4cd43b72009bb8bbe535a0c994803dff988d7a454eca0abf5af4e3a3bb747: Status 404 returned error can't find the container with id 65f4cd43b72009bb8bbe535a0c994803dff988d7a454eca0abf5af4e3a3bb747 Nov 03 20:25:58.513625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.513598 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5"] Nov 03 20:25:58.514803 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.514783 5111 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-cacerts\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.514914 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.514808 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9562k\" (UniqueName: \"kubernetes.io/projected/407008ca-4d98-4538-9f2c-bfdd9b03535a-kube-api-access-9562k\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.514914 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.514823 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.514914 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.514840 5111 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/407008ca-4d98-4538-9f2c-bfdd9b03535a-local-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.514914 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.514855 5111 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-csr-dns-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.514914 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.514868 5111 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/407008ca-4d98-4538-9f2c-bfdd9b03535a-istio-kubeconfig\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:58.688270 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.688232 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:25:58.688403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.688301 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" podUID="45038f2d-5a7b-4f31-92f3-147292e1c517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:58.752314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.752241 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" Nov 03 20:25:58.752314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.752276 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx" event={"ID":"45038f2d-5a7b-4f31-92f3-147292e1c517","Type":"ContainerDied","Data":"db1746b791feb57b0bde4ad68e2fe69ac6b4c80626acda5e90bb8fbd2c9cf330"} Nov 03 20:25:58.752600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.752348 5111 scope.go:122] "RemoveContainer" containerID="5fc029674e09a18493b9cfb5cda4b9322291acf45ce1dfcb5eb662eb7924a57c" Nov 03 20:25:58.754233 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.754206 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" Nov 03 20:25:58.754355 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.754237 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57" event={"ID":"7608cd01-af9c-4819-9016-6c010e0615ec","Type":"ContainerDied","Data":"971f3cf1cdf7b10d69531a0cdab496bd956cbf420d9002b53ebc8839dffde7a8"} Nov 03 20:25:58.756558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.756511 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" event={"ID":"407008ca-4d98-4538-9f2c-bfdd9b03535a","Type":"ContainerDied","Data":"abe5d9140d56cc65e3526a8c9482dd55d1e755f2cc2157d577363d2c75d34a4c"} Nov 03 20:25:58.756727 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.756711 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk" Nov 03 20:25:58.759964 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.759940 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5" event={"ID":"c61778c4-0880-4606-a5af-0ef731921ead","Type":"ContainerStarted","Data":"65f4cd43b72009bb8bbe535a0c994803dff988d7a454eca0abf5af4e3a3bb747"} Nov 03 20:25:58.761671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.761651 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" Nov 03 20:25:58.761767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.761676 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" event={"ID":"5d4d92cc-f4fa-421c-8b95-89838debd9b4","Type":"ContainerDied","Data":"f441a2f061da2eedc3ca6100fbf4d2fd740a34deade6b0531bd541c2a4cb3464"} Nov 03 20:25:58.768135 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.768118 5111 scope.go:122] "RemoveContainer" containerID="8025b778a3669dc573ea6a44cb67407245fc582d27f0148a58c14a7f7f174a11" Nov 03 20:25:58.770844 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.770824 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" Nov 03 20:25:58.770844 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.770834 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx" event={"ID":"95b31fd8-6624-466d-a5cf-2c84b5305b29","Type":"ContainerDied","Data":"cafb9029de22c718d6d509582cbd1e41a1ff44ad1bde495d55cff39287d5367f"} Nov 03 20:25:58.772504 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.772471 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb" event={"ID":"afe71f33-2cb2-437e-88b0-f544c9cbe663","Type":"ContainerStarted","Data":"d9df8bb1934327afd65e9e157428a54c87a13ac38955ceb687c55b66a468d6fc"} Nov 03 20:25:58.782359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.782334 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx"] Nov 03 20:25:58.788007 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.787898 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-pnvwx"] Nov 03 20:25:58.788119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.788101 5111 scope.go:122] "RemoveContainer" containerID="0c52f1466eeed8d7780f351a8661fd0c7b95439f189a7b03a7353411f5836ccf" Nov 03 20:25:58.799574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.799519 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57"] Nov 03 20:25:58.806994 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.806960 5111 scope.go:122] "RemoveContainer" containerID="e47df2ee3f51faaea58722cc8fe986a244c67c82605ea24fa0cf50399fbcda5d" Nov 03 20:25:58.810706 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.810677 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-8sg57"] Nov 03 20:25:58.824483 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.824458 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr"] Nov 03 20:25:58.831128 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.831101 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr"] Nov 03 20:25:58.835051 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.835031 5111 scope.go:122] "RemoveContainer" containerID="87087700fc59440bbde04edafa88932f94d7e8716e085e723f818a99710bcb45" Nov 03 20:25:58.836283 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.836263 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk"] Nov 03 20:25:58.840882 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.840861 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lfrxk"] Nov 03 20:25:58.851397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.851370 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx"] Nov 03 20:25:58.859107 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.857262 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-4wgbx"] Nov 03 20:25:58.984696 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.984662 5111 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:25:58.985121 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:58.984723 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-mkdnr" podUID="5d4d92cc-f4fa-421c-8b95-89838debd9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:59.827967 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:59.827866 5111 generic.go:356] "Generic (PLEG): container finished" podID="c61778c4-0880-4606-a5af-0ef731921ead" containerID="fc56f64b104ee650e80a41f60ca292e50dc0bd19369aeb275208c471f205715f" exitCode=0 Nov 03 20:25:59.827967 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:59.827943 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5" event={"ID":"c61778c4-0880-4606-a5af-0ef731921ead","Type":"ContainerDied","Data":"fc56f64b104ee650e80a41f60ca292e50dc0bd19369aeb275208c471f205715f"} Nov 03 20:25:59.834585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:59.832606 5111 generic.go:356] "Generic (PLEG): container finished" podID="afe71f33-2cb2-437e-88b0-f544c9cbe663" containerID="26543d232f342f8f6d18c7cccf4601473bdaa8ca8528076aafdb48f87ed4e1a1" exitCode=0 Nov 03 20:25:59.834585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:25:59.832758 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb" event={"ID":"afe71f33-2cb2-437e-88b0-f544c9cbe663","Type":"ContainerDied","Data":"26543d232f342f8f6d18c7cccf4601473bdaa8ca8528076aafdb48f87ed4e1a1"} Nov 03 20:26:00.188085 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:00.188051 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407008ca-4d98-4538-9f2c-bfdd9b03535a" path="/var/lib/kubelet/pods/407008ca-4d98-4538-9f2c-bfdd9b03535a/volumes" Nov 03 20:26:00.188562 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:00.188520 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45038f2d-5a7b-4f31-92f3-147292e1c517" path="/var/lib/kubelet/pods/45038f2d-5a7b-4f31-92f3-147292e1c517/volumes" Nov 03 20:26:00.189018 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:00.189001 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d4d92cc-f4fa-421c-8b95-89838debd9b4" path="/var/lib/kubelet/pods/5d4d92cc-f4fa-421c-8b95-89838debd9b4/volumes" Nov 03 20:26:00.189832 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:00.189810 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7608cd01-af9c-4819-9016-6c010e0615ec" path="/var/lib/kubelet/pods/7608cd01-af9c-4819-9016-6c010e0615ec/volumes" Nov 03 20:26:00.190257 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:00.190244 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b31fd8-6624-466d-a5cf-2c84b5305b29" path="/var/lib/kubelet/pods/95b31fd8-6624-466d-a5cf-2c84b5305b29/volumes" Nov 03 20:26:01.065334 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.065310 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb" Nov 03 20:26:01.067879 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.067858 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5" Nov 03 20:26:01.158788 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.158748 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afe71f33-2cb2-437e-88b0-f544c9cbe663-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe71f33-2cb2-437e-88b0-f544c9cbe663-maas-api-xks-ca\") pod \"afe71f33-2cb2-437e-88b0-f544c9cbe663\" (UID: \"afe71f33-2cb2-437e-88b0-f544c9cbe663\") " Nov 03 20:26:01.159014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.158813 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe71f33-2cb2-437e-88b0-f544c9cbe663-kube-api-access-rlp6k\" (UniqueName: \"kubernetes.io/projected/afe71f33-2cb2-437e-88b0-f544c9cbe663-kube-api-access-rlp6k\") pod \"afe71f33-2cb2-437e-88b0-f544c9cbe663\" (UID: \"afe71f33-2cb2-437e-88b0-f544c9cbe663\") " Nov 03 20:26:01.159014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.158854 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61778c4-0880-4606-a5af-0ef731921ead-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61778c4-0880-4606-a5af-0ef731921ead-maas-api-openshift-ca\") pod \"c61778c4-0880-4606-a5af-0ef731921ead\" (UID: \"c61778c4-0880-4606-a5af-0ef731921ead\") " Nov 03 20:26:01.159014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.158897 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe71f33-2cb2-437e-88b0-f544c9cbe663-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe71f33-2cb2-437e-88b0-f544c9cbe663-maas-api-openshift-ca\") pod \"afe71f33-2cb2-437e-88b0-f544c9cbe663\" (UID: \"afe71f33-2cb2-437e-88b0-f544c9cbe663\") " Nov 03 20:26:01.159014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.158951 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c61778c4-0880-4606-a5af-0ef731921ead-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61778c4-0880-4606-a5af-0ef731921ead-maas-api-xks-ca\") pod \"c61778c4-0880-4606-a5af-0ef731921ead\" (UID: \"c61778c4-0880-4606-a5af-0ef731921ead\") " Nov 03 20:26:01.159014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.158975 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61778c4-0880-4606-a5af-0ef731921ead-kube-api-access-444wb\" (UniqueName: \"kubernetes.io/projected/c61778c4-0880-4606-a5af-0ef731921ead-kube-api-access-444wb\") pod \"c61778c4-0880-4606-a5af-0ef731921ead\" (UID: \"c61778c4-0880-4606-a5af-0ef731921ead\") " Nov 03 20:26:01.163581 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.160522 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61778c4-0880-4606-a5af-0ef731921ead-maas-api-openshift-ca" pod "c61778c4-0880-4606-a5af-0ef731921ead" (UID: "c61778c4-0880-4606-a5af-0ef731921ead"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:01.163581 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.160827 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe71f33-2cb2-437e-88b0-f544c9cbe663-maas-api-openshift-ca" pod "afe71f33-2cb2-437e-88b0-f544c9cbe663" (UID: "afe71f33-2cb2-437e-88b0-f544c9cbe663"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:01.164708 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.164667 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe71f33-2cb2-437e-88b0-f544c9cbe663-maas-api-xks-ca" pod "afe71f33-2cb2-437e-88b0-f544c9cbe663" (UID: "afe71f33-2cb2-437e-88b0-f544c9cbe663"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:01.164708 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.164676 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c61778c4-0880-4606-a5af-0ef731921ead-maas-api-xks-ca" pod "c61778c4-0880-4606-a5af-0ef731921ead" (UID: "c61778c4-0880-4606-a5af-0ef731921ead"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:01.164881 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.164727 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61778c4-0880-4606-a5af-0ef731921ead-kube-api-access-444wb" pod "c61778c4-0880-4606-a5af-0ef731921ead" (UID: "c61778c4-0880-4606-a5af-0ef731921ead"). InnerVolumeSpecName "kube-api-access-444wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:01.165479 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.165459 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe71f33-2cb2-437e-88b0-f544c9cbe663-kube-api-access-rlp6k" pod "afe71f33-2cb2-437e-88b0-f544c9cbe663" (UID: "afe71f33-2cb2-437e-88b0-f544c9cbe663"). InnerVolumeSpecName "kube-api-access-rlp6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:01.261510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.261477 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61778c4-0880-4606-a5af-0ef731921ead-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:01.261510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.261507 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-444wb\" (UniqueName: \"kubernetes.io/projected/c61778c4-0880-4606-a5af-0ef731921ead-kube-api-access-444wb\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:01.261510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.261517 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe71f33-2cb2-437e-88b0-f544c9cbe663-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:01.262001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.261526 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlp6k\" (UniqueName: \"kubernetes.io/projected/afe71f33-2cb2-437e-88b0-f544c9cbe663-kube-api-access-rlp6k\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:01.262001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.261571 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61778c4-0880-4606-a5af-0ef731921ead-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:01.262001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.261584 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe71f33-2cb2-437e-88b0-f544c9cbe663-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:01.641462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.641433 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:26:01.773690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.773613 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-podinfo\") pod \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " Nov 03 20:26:01.773690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.773653 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-workload-certs\") pod \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " Nov 03 20:26:01.773690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.773675 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-token\" (UniqueName: \"kubernetes.io/projected/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-token\") pod \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " Nov 03 20:26:01.773959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.773738 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istiod-ca-cert\") pod \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " Nov 03 20:26:01.773959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.773817 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-envoy\") pod \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " Nov 03 20:26:01.773959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.773861 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-data\") pod \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " Nov 03 20:26:01.774115 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.773973 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-workload-certs" pod "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" (UID: "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:01.774115 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.773995 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-credential-socket\") pod \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " Nov 03 20:26:01.774115 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.774069 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-kube-api-access-zqbzn\" (UniqueName: \"kubernetes.io/projected/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-kube-api-access-zqbzn\") pod \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " Nov 03 20:26:01.774115 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.774108 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-workload-socket\") pod \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\" (UID: \"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c\") " Nov 03 20:26:01.774331 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.774206 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istiod-ca-cert" pod "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" (UID: "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:01.774331 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.774292 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-credential-socket" pod "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" (UID: "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:01.775078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.774437 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-data" pod "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" (UID: "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:01.775078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.774483 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-workload-socket" pod "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" (UID: "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:01.775078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.774982 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:01.775078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.775008 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:01.775078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.775024 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:01.775078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.775038 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:01.775078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.775053 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:01.776212 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.776192 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-envoy" pod "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" (UID: "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:01.776297 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.776263 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-podinfo" pod "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" (UID: "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:01.776447 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.776433 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-kube-api-access-zqbzn" pod "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" (UID: "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c"). InnerVolumeSpecName "kube-api-access-zqbzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:01.776505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.776455 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-token" pod "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" (UID: "6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:01.852520 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.852483 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5" Nov 03 20:26:01.852708 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.852521 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5" event={"ID":"c61778c4-0880-4606-a5af-0ef731921ead","Type":"ContainerDied","Data":"65f4cd43b72009bb8bbe535a0c994803dff988d7a454eca0abf5af4e3a3bb747"} Nov 03 20:26:01.852708 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.852591 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65f4cd43b72009bb8bbe535a0c994803dff988d7a454eca0abf5af4e3a3bb747" Nov 03 20:26:01.859303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.859266 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb" Nov 03 20:26:01.859303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.859287 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb" event={"ID":"afe71f33-2cb2-437e-88b0-f544c9cbe663","Type":"ContainerDied","Data":"d9df8bb1934327afd65e9e157428a54c87a13ac38955ceb687c55b66a468d6fc"} Nov 03 20:26:01.859521 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.859385 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9df8bb1934327afd65e9e157428a54c87a13ac38955ceb687c55b66a468d6fc" Nov 03 20:26:01.861278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.861246 5111 generic.go:356] "Generic (PLEG): container finished" podID="6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" containerID="6ed9c0d6de305e562e5855fbee5080107934d155639e2801bde5743c7c659ef5" exitCode=0 Nov 03 20:26:01.861442 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.861422 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" event={"ID":"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c","Type":"ContainerDied","Data":"6ed9c0d6de305e562e5855fbee5080107934d155639e2801bde5743c7c659ef5"} Nov 03 20:26:01.861511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.861494 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" event={"ID":"6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c","Type":"ContainerDied","Data":"136e2b3a6877d53248f60d2834328f972b8f891b0874f1e1d08fa2244898da61"} Nov 03 20:26:01.861602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.861553 5111 scope.go:122] "RemoveContainer" containerID="6ed9c0d6de305e562e5855fbee5080107934d155639e2801bde5743c7c659ef5" Nov 03 20:26:01.861779 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.861761 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl" Nov 03 20:26:01.876545 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.876293 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqbzn\" (UniqueName: \"kubernetes.io/projected/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-kube-api-access-zqbzn\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:01.876545 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.876323 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:01.876545 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.876339 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:01.876545 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.876353 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:01.885037 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.885008 5111 scope.go:122] "RemoveContainer" containerID="6ed9c0d6de305e562e5855fbee5080107934d155639e2801bde5743c7c659ef5" Nov 03 20:26:01.885375 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:01.885351 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ed9c0d6de305e562e5855fbee5080107934d155639e2801bde5743c7c659ef5\": container with ID starting with 6ed9c0d6de305e562e5855fbee5080107934d155639e2801bde5743c7c659ef5 not found: ID does not exist" containerID="6ed9c0d6de305e562e5855fbee5080107934d155639e2801bde5743c7c659ef5" Nov 03 20:26:01.885447 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.885387 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ed9c0d6de305e562e5855fbee5080107934d155639e2801bde5743c7c659ef5"} err="failed to get container status \"6ed9c0d6de305e562e5855fbee5080107934d155639e2801bde5743c7c659ef5\": rpc error: code = NotFound desc = could not find container \"6ed9c0d6de305e562e5855fbee5080107934d155639e2801bde5743c7c659ef5\": container with ID starting with 6ed9c0d6de305e562e5855fbee5080107934d155639e2801bde5743c7c659ef5 not found: ID does not exist" Nov 03 20:26:01.898317 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.898283 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl"] Nov 03 20:26:01.902510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:01.902479 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-rsjjl"] Nov 03 20:26:02.188234 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:02.188197 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" path="/var/lib/kubelet/pods/6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c/volumes" Nov 03 20:26:02.851758 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:02.851705 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5"] Nov 03 20:26:03.986902 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:03.977632 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh"] Nov 03 20:26:03.991810 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:03.991765 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0ffd89-8j5mh"] Nov 03 20:26:04.193590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:04.193194 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0947ae5b-4856-4c5a-88dc-7ffc8321b06f" path="/var/lib/kubelet/pods/0947ae5b-4856-4c5a-88dc-7ffc8321b06f/volumes" Nov 03 20:26:05.096661 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:05.096623 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:26:06.536804 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:06.536767 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr"] Nov 03 20:26:06.537779 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:06.537720 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" podUID="28da3b98-e5f9-4355-876f-560f633061de" containerName="istio-proxy" containerID="cri-o://308dfd08165678b1b04a95906e90078ecdbf6c87a114bfc8e4ebd0c4b608d59a" gracePeriod=30 Nov 03 20:26:06.552056 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:06.552026 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s"] Nov 03 20:26:06.552246 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:06.552226 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" podUID="3563a606-4dee-47cd-815f-ace2af65416b" containerName="istio-proxy" containerID="cri-o://5d9eef1047bf66d368a9a2e1b58a1a8a6b349ccbabb3e5a928fcae319695d96e" gracePeriod=30 Nov 03 20:26:07.013319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.013279 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m"] Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014374 5111 cpu_manager.go:414] "containerMap: removing container" podUID="95b31fd8-6624-466d-a5cf-2c84b5305b29" containerName="istio-proxy" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014395 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b31fd8-6624-466d-a5cf-2c84b5305b29" containerName="istio-proxy" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014414 5111 cpu_manager.go:414] "containerMap: removing container" podUID="7608cd01-af9c-4819-9016-6c010e0615ec" containerName="istio-proxy" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014422 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7608cd01-af9c-4819-9016-6c010e0615ec" containerName="istio-proxy" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014452 5111 cpu_manager.go:414] "containerMap: removing container" podUID="5d4d92cc-f4fa-421c-8b95-89838debd9b4" containerName="istio-proxy" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014460 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d4d92cc-f4fa-421c-8b95-89838debd9b4" containerName="istio-proxy" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014474 5111 cpu_manager.go:414] "containerMap: removing container" podUID="0947ae5b-4856-4c5a-88dc-7ffc8321b06f" containerName="revoke-keys" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014482 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0947ae5b-4856-4c5a-88dc-7ffc8321b06f" containerName="revoke-keys" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014514 5111 cpu_manager.go:414] "containerMap: removing container" podUID="45038f2d-5a7b-4f31-92f3-147292e1c517" containerName="istio-proxy" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014521 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45038f2d-5a7b-4f31-92f3-147292e1c517" containerName="istio-proxy" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014549 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" containerName="istio-proxy" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014559 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" containerName="istio-proxy" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014753 5111 cpu_manager.go:414] "containerMap: removing container" podUID="afe71f33-2cb2-437e-88b0-f544c9cbe663" containerName="revoke-keys" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014764 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe71f33-2cb2-437e-88b0-f544c9cbe663" containerName="revoke-keys" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014782 5111 cpu_manager.go:414] "containerMap: removing container" podUID="407008ca-4d98-4538-9f2c-bfdd9b03535a" containerName="discovery" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014791 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407008ca-4d98-4538-9f2c-bfdd9b03535a" containerName="discovery" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014811 5111 cpu_manager.go:414] "containerMap: removing container" podUID="407c2001-b858-4378-a649-c300979e4f85" containerName="payload-pre-processing" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014820 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407c2001-b858-4378-a649-c300979e4f85" containerName="payload-pre-processing" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014830 5111 cpu_manager.go:414] "containerMap: removing container" podUID="2aa57a2e-9b53-4c9e-a17c-48f12faaac69" containerName="payload-processing" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014838 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa57a2e-9b53-4c9e-a17c-48f12faaac69" containerName="payload-processing" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014855 5111 cpu_manager.go:414] "containerMap: removing container" podUID="c61778c4-0880-4606-a5af-0ef731921ead" containerName="revoke-keys" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.014863 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61778c4-0880-4606-a5af-0ef731921ead" containerName="revoke-keys" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.015066 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="45038f2d-5a7b-4f31-92f3-147292e1c517" containerName="istio-proxy" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.015081 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f56c4a1-e91a-4200-ae75-a2f91cc8ac0c" containerName="istio-proxy" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.015094 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe71f33-2cb2-437e-88b0-f544c9cbe663" containerName="revoke-keys" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.015105 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="407008ca-4d98-4538-9f2c-bfdd9b03535a" containerName="discovery" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.015116 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="407c2001-b858-4378-a649-c300979e4f85" containerName="payload-pre-processing" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.015127 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa57a2e-9b53-4c9e-a17c-48f12faaac69" containerName="payload-processing" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.015137 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61778c4-0880-4606-a5af-0ef731921ead" containerName="revoke-keys" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.015152 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b31fd8-6624-466d-a5cf-2c84b5305b29" containerName="istio-proxy" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.015162 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="7608cd01-af9c-4819-9016-6c010e0615ec" containerName="istio-proxy" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.015175 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d4d92cc-f4fa-421c-8b95-89838debd9b4" containerName="istio-proxy" Nov 03 20:26:07.015359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.015184 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="0947ae5b-4856-4c5a-88dc-7ffc8321b06f" containerName="revoke-keys" Nov 03 20:26:07.023280 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.023045 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m" Nov 03 20:26:07.027412 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.027381 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m"] Nov 03 20:26:07.033644 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.031472 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:26:07.033644 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.032479 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:26:07.096302 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.096268 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz"] Nov 03 20:26:07.106910 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.106870 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz"] Nov 03 20:26:07.107073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.106966 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" Nov 03 20:26:07.130383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.130350 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt87l\" (UniqueName: \"kubernetes.io/projected/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-kube-api-access-bt87l\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m\" (UID: \"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m" Nov 03 20:26:07.130556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.130510 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m\" (UID: \"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m" Nov 03 20:26:07.130731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.130711 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m\" (UID: \"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m" Nov 03 20:26:07.232281 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.232242 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/873efceb-087b-4442-966e-e395b73ae3c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e63e2422-2zcbz\" (UID: \"873efceb-087b-4442-966e-e395b73ae3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" Nov 03 20:26:07.232776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.232747 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m\" (UID: \"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m" Nov 03 20:26:07.233018 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.233001 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt87l\" (UniqueName: \"kubernetes.io/projected/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-kube-api-access-bt87l\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m\" (UID: \"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m" Nov 03 20:26:07.233112 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.233044 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99jhh\" (UniqueName: \"kubernetes.io/projected/873efceb-087b-4442-966e-e395b73ae3c5-kube-api-access-99jhh\") pod \"maas-api-revoke-sub-e63e2422-2zcbz\" (UID: \"873efceb-087b-4442-966e-e395b73ae3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" Nov 03 20:26:07.233169 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.233118 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/873efceb-087b-4442-966e-e395b73ae3c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e63e2422-2zcbz\" (UID: \"873efceb-087b-4442-966e-e395b73ae3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" Nov 03 20:26:07.233265 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.233241 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m\" (UID: \"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m" Nov 03 20:26:07.233573 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.233521 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m\" (UID: \"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m" Nov 03 20:26:07.235844 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.235813 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m\" (UID: \"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m" Nov 03 20:26:07.246914 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.246883 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8"] Nov 03 20:26:07.247326 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.247304 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt87l\" (UniqueName: \"kubernetes.io/projected/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-kube-api-access-bt87l\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m\" (UID: \"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m" Nov 03 20:26:07.335310 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.335214 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99jhh\" (UniqueName: \"kubernetes.io/projected/873efceb-087b-4442-966e-e395b73ae3c5-kube-api-access-99jhh\") pod \"maas-api-revoke-sub-e63e2422-2zcbz\" (UID: \"873efceb-087b-4442-966e-e395b73ae3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" Nov 03 20:26:07.335310 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.335298 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/873efceb-087b-4442-966e-e395b73ae3c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e63e2422-2zcbz\" (UID: \"873efceb-087b-4442-966e-e395b73ae3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" Nov 03 20:26:07.335627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.335427 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/873efceb-087b-4442-966e-e395b73ae3c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e63e2422-2zcbz\" (UID: \"873efceb-087b-4442-966e-e395b73ae3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" Nov 03 20:26:07.336309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.336280 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/873efceb-087b-4442-966e-e395b73ae3c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e63e2422-2zcbz\" (UID: \"873efceb-087b-4442-966e-e395b73ae3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" Nov 03 20:26:07.339429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.339400 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/873efceb-087b-4442-966e-e395b73ae3c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e63e2422-2zcbz\" (UID: \"873efceb-087b-4442-966e-e395b73ae3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" Nov 03 20:26:07.347273 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.347244 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99jhh\" (UniqueName: \"kubernetes.io/projected/873efceb-087b-4442-966e-e395b73ae3c5-kube-api-access-99jhh\") pod \"maas-api-revoke-sub-e63e2422-2zcbz\" (UID: \"873efceb-087b-4442-966e-e395b73ae3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" Nov 03 20:26:07.349734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.349710 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m" Nov 03 20:26:07.436911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.426641 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" Nov 03 20:26:07.852371 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.852323 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" podUID="32f418b9-ac9f-4c9f-bac7-7a3341df7747" containerName="payload-processing" containerID="cri-o://e4a0dba604a5141fefe476e63f1be163fcc025c8c6a79731bfa45863faac54d0" gracePeriod=25 Nov 03 20:26:07.856253 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.856222 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m"] Nov 03 20:26:07.856837 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:26:07.856808 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbfe191_1dfa_4dd3_9d6c_8bbe7637a6b7.slice/crio-6828963044d9d0a5507f4032fc4e5a145ec8503a28bdfaaf3b974c0b1b454120 WatchSource:0}: Error finding container 6828963044d9d0a5507f4032fc4e5a145ec8503a28bdfaaf3b974c0b1b454120: Status 404 returned error can't find the container with id 6828963044d9d0a5507f4032fc4e5a145ec8503a28bdfaaf3b974c0b1b454120 Nov 03 20:26:07.895776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:07.895748 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz"] Nov 03 20:26:07.916568 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:26:07.916526 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873efceb_087b_4442_966e_e395b73ae3c5.slice/crio-1dbe1721f86230c86482e04bd922ea2d09a39b99e62fc30cb1e9b9fb2cdb8f3a WatchSource:0}: Error finding container 1dbe1721f86230c86482e04bd922ea2d09a39b99e62fc30cb1e9b9fb2cdb8f3a: Status 404 returned error can't find the container with id 1dbe1721f86230c86482e04bd922ea2d09a39b99e62fc30cb1e9b9fb2cdb8f3a Nov 03 20:26:08.005625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:08.005590 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m" event={"ID":"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7","Type":"ContainerStarted","Data":"6828963044d9d0a5507f4032fc4e5a145ec8503a28bdfaaf3b974c0b1b454120"} Nov 03 20:26:08.007659 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:08.007626 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" event={"ID":"32f418b9-ac9f-4c9f-bac7-7a3341df7747","Type":"ContainerDied","Data":"e4a0dba604a5141fefe476e63f1be163fcc025c8c6a79731bfa45863faac54d0"} Nov 03 20:26:08.007796 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:08.007698 5111 generic.go:356] "Generic (PLEG): container finished" podID="32f418b9-ac9f-4c9f-bac7-7a3341df7747" containerID="e4a0dba604a5141fefe476e63f1be163fcc025c8c6a79731bfa45863faac54d0" exitCode=0 Nov 03 20:26:08.009644 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:08.009613 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" event={"ID":"873efceb-087b-4442-966e-e395b73ae3c5","Type":"ContainerStarted","Data":"1dbe1721f86230c86482e04bd922ea2d09a39b99e62fc30cb1e9b9fb2cdb8f3a"} Nov 03 20:26:08.297659 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:08.297625 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" Nov 03 20:26:08.436712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:08.436625 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32f418b9-ac9f-4c9f-bac7-7a3341df7747-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32f418b9-ac9f-4c9f-bac7-7a3341df7747-plugins-config-volume\") pod \"32f418b9-ac9f-4c9f-bac7-7a3341df7747\" (UID: \"32f418b9-ac9f-4c9f-bac7-7a3341df7747\") " Nov 03 20:26:08.436883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:08.436782 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f418b9-ac9f-4c9f-bac7-7a3341df7747-kube-api-access-wtfl6\" (UniqueName: \"kubernetes.io/projected/32f418b9-ac9f-4c9f-bac7-7a3341df7747-kube-api-access-wtfl6\") pod \"32f418b9-ac9f-4c9f-bac7-7a3341df7747\" (UID: \"32f418b9-ac9f-4c9f-bac7-7a3341df7747\") " Nov 03 20:26:08.437027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:08.437006 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f418b9-ac9f-4c9f-bac7-7a3341df7747-plugins-config-volume" pod "32f418b9-ac9f-4c9f-bac7-7a3341df7747" (UID: "32f418b9-ac9f-4c9f-bac7-7a3341df7747"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:08.437464 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:08.437447 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32f418b9-ac9f-4c9f-bac7-7a3341df7747-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:08.439147 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:08.439118 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f418b9-ac9f-4c9f-bac7-7a3341df7747-kube-api-access-wtfl6" pod "32f418b9-ac9f-4c9f-bac7-7a3341df7747" (UID: "32f418b9-ac9f-4c9f-bac7-7a3341df7747"). InnerVolumeSpecName "kube-api-access-wtfl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:08.538545 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:08.538507 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtfl6\" (UniqueName: \"kubernetes.io/projected/32f418b9-ac9f-4c9f-bac7-7a3341df7747-kube-api-access-wtfl6\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:09.020297 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:09.020262 5111 generic.go:356] "Generic (PLEG): container finished" podID="bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7" containerID="5dc0dba587c8b92f4e91c6640c70ee9623857f1b82bfd6001753c0b73203233e" exitCode=0 Nov 03 20:26:09.020762 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:09.020352 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m" event={"ID":"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7","Type":"ContainerDied","Data":"5dc0dba587c8b92f4e91c6640c70ee9623857f1b82bfd6001753c0b73203233e"} Nov 03 20:26:09.021962 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:09.021942 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" Nov 03 20:26:09.021962 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:09.021951 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5" event={"ID":"32f418b9-ac9f-4c9f-bac7-7a3341df7747","Type":"ContainerDied","Data":"039a438d047626174a825391d1da56dfdfc8356a1d6c20f9a0c140803e41a3e9"} Nov 03 20:26:09.022166 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:09.022003 5111 scope.go:122] "RemoveContainer" containerID="e4a0dba604a5141fefe476e63f1be163fcc025c8c6a79731bfa45863faac54d0" Nov 03 20:26:09.024159 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:09.024126 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" event={"ID":"873efceb-087b-4442-966e-e395b73ae3c5","Type":"ContainerStarted","Data":"731328f80df547b67c70f18d464eaa344a28d2bb0a6d5148155308ed44eb4c0a"} Nov 03 20:26:09.061360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:09.061307 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" podStartSLOduration=2.061293566 podStartE2EDuration="2.061293566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:09.058426066 +0000 UTC m=+2799.020804505" watchObservedRunningTime="2026-11-03 20:26:09.061293566 +0000 UTC m=+2799.023672003" Nov 03 20:26:09.077069 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:09.077038 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5"] Nov 03 20:26:09.080926 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:09.080902 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0ffd89-7d7dc89d8d-45gj5"] Nov 03 20:26:09.418953 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:09.418908 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx"] Nov 03 20:26:09.419149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:09.419123 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" podUID="97388cfd-a7f0-425d-946b-bf0181f548b0" containerName="istio-proxy" containerID="cri-o://49a572d4425500309fe26eac2e1d22038f7c103faee0d2fce8204dec65f313f5" gracePeriod=30 Nov 03 20:26:10.039401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.039363 5111 generic.go:356] "Generic (PLEG): container finished" podID="873efceb-087b-4442-966e-e395b73ae3c5" containerID="731328f80df547b67c70f18d464eaa344a28d2bb0a6d5148155308ed44eb4c0a" exitCode=0 Nov 03 20:26:10.039931 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.039445 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" event={"ID":"873efceb-087b-4442-966e-e395b73ae3c5","Type":"ContainerDied","Data":"731328f80df547b67c70f18d464eaa344a28d2bb0a6d5148155308ed44eb4c0a"} Nov 03 20:26:10.074274 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.074238 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj"] Nov 03 20:26:10.074500 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.074471 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" podUID="8edc2022-355d-4c8e-9564-f2c0a2c6b397" containerName="istio-proxy" containerID="cri-o://6bca2a51b6953dbbd9c11fd32777b34f6b955e3ff0dd1b6761d2dd5a21d4b555" gracePeriod=30 Nov 03 20:26:10.193053 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.193021 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f418b9-ac9f-4c9f-bac7-7a3341df7747" path="/var/lib/kubelet/pods/32f418b9-ac9f-4c9f-bac7-7a3341df7747/volumes" Nov 03 20:26:10.273905 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.273874 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m" Nov 03 20:26:10.371342 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.371312 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-maas-api-openshift-ca\") pod \"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7\" (UID: \"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7\") " Nov 03 20:26:10.371504 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.371415 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-kube-api-access-bt87l\" (UniqueName: \"kubernetes.io/projected/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-kube-api-access-bt87l\") pod \"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7\" (UID: \"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7\") " Nov 03 20:26:10.371504 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.371484 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-maas-api-xks-ca\") pod \"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7\" (UID: \"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7\") " Nov 03 20:26:10.371714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.371690 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-maas-api-openshift-ca" pod "bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7" (UID: "bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:10.372041 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.372011 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:10.373932 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.373909 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-kube-api-access-bt87l" pod "bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7" (UID: "bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7"). InnerVolumeSpecName "kube-api-access-bt87l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:10.373932 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.373920 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-maas-api-xks-ca" pod "bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7" (UID: "bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:10.473111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.473080 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt87l\" (UniqueName: \"kubernetes.io/projected/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-kube-api-access-bt87l\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:10.473111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.473110 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:10.817242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.817146 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v"] Nov 03 20:26:10.817474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:10.817440 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" podUID="1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" containerName="istio-proxy" containerID="cri-o://54d095a0b7b62c3ae44ba288a768ae9a5ec315a2df862692323fa8e00c2a468e" gracePeriod=30 Nov 03 20:26:11.072813 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.072431 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m" Nov 03 20:26:11.072813 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.072612 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m" event={"ID":"bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7","Type":"ContainerDied","Data":"6828963044d9d0a5507f4032fc4e5a145ec8503a28bdfaaf3b974c0b1b454120"} Nov 03 20:26:11.072813 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.072662 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6828963044d9d0a5507f4032fc4e5a145ec8503a28bdfaaf3b974c0b1b454120" Nov 03 20:26:11.245493 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.245468 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" Nov 03 20:26:11.286979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.286945 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/873efceb-087b-4442-966e-e395b73ae3c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/873efceb-087b-4442-966e-e395b73ae3c5-maas-api-xks-ca\") pod \"873efceb-087b-4442-966e-e395b73ae3c5\" (UID: \"873efceb-087b-4442-966e-e395b73ae3c5\") " Nov 03 20:26:11.287160 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.287009 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/873efceb-087b-4442-966e-e395b73ae3c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/873efceb-087b-4442-966e-e395b73ae3c5-maas-api-openshift-ca\") pod \"873efceb-087b-4442-966e-e395b73ae3c5\" (UID: \"873efceb-087b-4442-966e-e395b73ae3c5\") " Nov 03 20:26:11.287160 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.287134 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/873efceb-087b-4442-966e-e395b73ae3c5-kube-api-access-99jhh\" (UniqueName: \"kubernetes.io/projected/873efceb-087b-4442-966e-e395b73ae3c5-kube-api-access-99jhh\") pod \"873efceb-087b-4442-966e-e395b73ae3c5\" (UID: \"873efceb-087b-4442-966e-e395b73ae3c5\") " Nov 03 20:26:11.287501 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.287459 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/873efceb-087b-4442-966e-e395b73ae3c5-maas-api-openshift-ca" pod "873efceb-087b-4442-966e-e395b73ae3c5" (UID: "873efceb-087b-4442-966e-e395b73ae3c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:11.287897 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.287871 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/873efceb-087b-4442-966e-e395b73ae3c5-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:11.289326 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.289295 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/873efceb-087b-4442-966e-e395b73ae3c5-maas-api-xks-ca" pod "873efceb-087b-4442-966e-e395b73ae3c5" (UID: "873efceb-087b-4442-966e-e395b73ae3c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:11.289597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.289512 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873efceb-087b-4442-966e-e395b73ae3c5-kube-api-access-99jhh" pod "873efceb-087b-4442-966e-e395b73ae3c5" (UID: "873efceb-087b-4442-966e-e395b73ae3c5"). InnerVolumeSpecName "kube-api-access-99jhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:11.389688 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.389652 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/873efceb-087b-4442-966e-e395b73ae3c5-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:11.389688 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.389688 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99jhh\" (UniqueName: \"kubernetes.io/projected/873efceb-087b-4442-966e-e395b73ae3c5-kube-api-access-99jhh\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:11.692204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.692126 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z"] Nov 03 20:26:11.693332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.693302 5111 cpu_manager.go:414] "containerMap: removing container" podUID="873efceb-087b-4442-966e-e395b73ae3c5" containerName="revoke-keys" Nov 03 20:26:11.693332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.693332 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="873efceb-087b-4442-966e-e395b73ae3c5" containerName="revoke-keys" Nov 03 20:26:11.693582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.693352 5111 cpu_manager.go:414] "containerMap: removing container" podUID="bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7" containerName="revoke-keys" Nov 03 20:26:11.693582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.693360 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7" containerName="revoke-keys" Nov 03 20:26:11.693582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.693416 5111 cpu_manager.go:414] "containerMap: removing container" podUID="32f418b9-ac9f-4c9f-bac7-7a3341df7747" containerName="payload-processing" Nov 03 20:26:11.693582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.693424 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f418b9-ac9f-4c9f-bac7-7a3341df7747" containerName="payload-processing" Nov 03 20:26:11.693776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.693606 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="873efceb-087b-4442-966e-e395b73ae3c5" containerName="revoke-keys" Nov 03 20:26:11.693776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.693622 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7" containerName="revoke-keys" Nov 03 20:26:11.693776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.693640 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f418b9-ac9f-4c9f-bac7-7a3341df7747" containerName="payload-processing" Nov 03 20:26:11.701952 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.701922 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" Nov 03 20:26:11.704207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.704182 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z"] Nov 03 20:26:11.795789 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.795755 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z\" (UID: \"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" Nov 03 20:26:11.796124 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.795805 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z\" (UID: \"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" Nov 03 20:26:11.796124 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.796039 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdmxb\" (UniqueName: \"kubernetes.io/projected/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-kube-api-access-qdmxb\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z\" (UID: \"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" Nov 03 20:26:11.899598 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.898044 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z\" (UID: \"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" Nov 03 20:26:11.907148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.900344 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z\" (UID: \"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" Nov 03 20:26:11.907148 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.900470 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdmxb\" (UniqueName: \"kubernetes.io/projected/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-kube-api-access-qdmxb\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z\" (UID: \"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" Nov 03 20:26:11.909923 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.909815 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z\" (UID: \"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" Nov 03 20:26:11.916912 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.915906 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z\" (UID: \"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" Nov 03 20:26:11.918058 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.917887 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdmxb\" (UniqueName: \"kubernetes.io/projected/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-kube-api-access-qdmxb\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z\" (UID: \"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" Nov 03 20:26:11.919183 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.919146 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:26:11.953067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:11.953033 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:26:12.001736 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.001701 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3563a606-4dee-47cd-815f-ace2af65416b-istio-token\" (UniqueName: \"kubernetes.io/projected/3563a606-4dee-47cd-815f-ace2af65416b-istio-token\") pod \"3563a606-4dee-47cd-815f-ace2af65416b\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " Nov 03 20:26:12.001950 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.001759 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3563a606-4dee-47cd-815f-ace2af65416b-kube-api-access-fdrpf\" (UniqueName: \"kubernetes.io/projected/3563a606-4dee-47cd-815f-ace2af65416b-kube-api-access-fdrpf\") pod \"3563a606-4dee-47cd-815f-ace2af65416b\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " Nov 03 20:26:12.001950 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.001876 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-istio-envoy\") pod \"28da3b98-e5f9-4355-876f-560f633061de\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " Nov 03 20:26:12.001950 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.001930 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-workload-socket\") pod \"3563a606-4dee-47cd-815f-ace2af65416b\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " Nov 03 20:26:12.002139 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.001965 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28da3b98-e5f9-4355-876f-560f633061de-istio-token\" (UniqueName: \"kubernetes.io/projected/28da3b98-e5f9-4355-876f-560f633061de-istio-token\") pod \"28da3b98-e5f9-4355-876f-560f633061de\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " Nov 03 20:26:12.002139 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.002069 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28da3b98-e5f9-4355-876f-560f633061de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28da3b98-e5f9-4355-876f-560f633061de-istio-podinfo\") pod \"28da3b98-e5f9-4355-876f-560f633061de\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " Nov 03 20:26:12.002139 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.002122 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-credential-socket\") pod \"3563a606-4dee-47cd-815f-ace2af65416b\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " Nov 03 20:26:12.002302 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.002190 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-credential-socket\") pod \"28da3b98-e5f9-4355-876f-560f633061de\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " Nov 03 20:26:12.002302 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.002237 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-istio-envoy\") pod \"3563a606-4dee-47cd-815f-ace2af65416b\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " Nov 03 20:26:12.002420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.002311 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-workload-certs\") pod \"3563a606-4dee-47cd-815f-ace2af65416b\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " Nov 03 20:26:12.002420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.002344 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-workload-certs\") pod \"28da3b98-e5f9-4355-876f-560f633061de\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " Nov 03 20:26:12.002420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.002377 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3563a606-4dee-47cd-815f-ace2af65416b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3563a606-4dee-47cd-815f-ace2af65416b-istiod-ca-cert\") pod \"3563a606-4dee-47cd-815f-ace2af65416b\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " Nov 03 20:26:12.002599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.002435 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-istio-data\") pod \"28da3b98-e5f9-4355-876f-560f633061de\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " Nov 03 20:26:12.002599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.002440 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-workload-socket" pod "3563a606-4dee-47cd-815f-ace2af65416b" (UID: "3563a606-4dee-47cd-815f-ace2af65416b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:12.002599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.002473 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-istio-data\") pod \"3563a606-4dee-47cd-815f-ace2af65416b\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " Nov 03 20:26:12.002599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.002511 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3563a606-4dee-47cd-815f-ace2af65416b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3563a606-4dee-47cd-815f-ace2af65416b-istio-podinfo\") pod \"3563a606-4dee-47cd-815f-ace2af65416b\" (UID: \"3563a606-4dee-47cd-815f-ace2af65416b\") " Nov 03 20:26:12.002599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.002562 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28da3b98-e5f9-4355-876f-560f633061de-kube-api-access-fddjp\" (UniqueName: \"kubernetes.io/projected/28da3b98-e5f9-4355-876f-560f633061de-kube-api-access-fddjp\") pod \"28da3b98-e5f9-4355-876f-560f633061de\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " Nov 03 20:26:12.002905 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.002632 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-workload-socket\") pod \"28da3b98-e5f9-4355-876f-560f633061de\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " Nov 03 20:26:12.002905 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.002659 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28da3b98-e5f9-4355-876f-560f633061de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28da3b98-e5f9-4355-876f-560f633061de-istiod-ca-cert\") pod \"28da3b98-e5f9-4355-876f-560f633061de\" (UID: \"28da3b98-e5f9-4355-876f-560f633061de\") " Nov 03 20:26:12.003443 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.003418 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.003842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.003805 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28da3b98-e5f9-4355-876f-560f633061de-istiod-ca-cert" pod "28da3b98-e5f9-4355-876f-560f633061de" (UID: "28da3b98-e5f9-4355-876f-560f633061de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:12.004119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.004093 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-workload-certs" pod "3563a606-4dee-47cd-815f-ace2af65416b" (UID: "3563a606-4dee-47cd-815f-ace2af65416b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:12.004324 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.004303 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-workload-certs" pod "28da3b98-e5f9-4355-876f-560f633061de" (UID: "28da3b98-e5f9-4355-876f-560f633061de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:12.004642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.004617 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3563a606-4dee-47cd-815f-ace2af65416b-istiod-ca-cert" pod "3563a606-4dee-47cd-815f-ace2af65416b" (UID: "3563a606-4dee-47cd-815f-ace2af65416b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:12.004888 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.004865 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-istio-data" pod "28da3b98-e5f9-4355-876f-560f633061de" (UID: "28da3b98-e5f9-4355-876f-560f633061de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:12.005088 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.005067 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-istio-data" pod "3563a606-4dee-47cd-815f-ace2af65416b" (UID: "3563a606-4dee-47cd-815f-ace2af65416b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:12.005705 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.005677 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-istio-envoy" pod "3563a606-4dee-47cd-815f-ace2af65416b" (UID: "3563a606-4dee-47cd-815f-ace2af65416b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:12.006734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.006491 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-workload-socket" pod "28da3b98-e5f9-4355-876f-560f633061de" (UID: "28da3b98-e5f9-4355-876f-560f633061de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:12.007049 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.006873 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-credential-socket" pod "3563a606-4dee-47cd-815f-ace2af65416b" (UID: "3563a606-4dee-47cd-815f-ace2af65416b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:12.007221 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.007186 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-credential-socket" pod "28da3b98-e5f9-4355-876f-560f633061de" (UID: "28da3b98-e5f9-4355-876f-560f633061de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:12.007709 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.007672 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28da3b98-e5f9-4355-876f-560f633061de-istio-podinfo" pod "28da3b98-e5f9-4355-876f-560f633061de" (UID: "28da3b98-e5f9-4355-876f-560f633061de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:12.008460 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.008425 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28da3b98-e5f9-4355-876f-560f633061de-kube-api-access-fddjp" pod "28da3b98-e5f9-4355-876f-560f633061de" (UID: "28da3b98-e5f9-4355-876f-560f633061de"). InnerVolumeSpecName "kube-api-access-fddjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:12.009468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.009434 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28da3b98-e5f9-4355-876f-560f633061de-istio-token" pod "28da3b98-e5f9-4355-876f-560f633061de" (UID: "28da3b98-e5f9-4355-876f-560f633061de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:12.009610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.009563 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3563a606-4dee-47cd-815f-ace2af65416b-istio-podinfo" pod "3563a606-4dee-47cd-815f-ace2af65416b" (UID: "3563a606-4dee-47cd-815f-ace2af65416b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:12.015446 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.015413 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3563a606-4dee-47cd-815f-ace2af65416b-istio-token" pod "3563a606-4dee-47cd-815f-ace2af65416b" (UID: "3563a606-4dee-47cd-815f-ace2af65416b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:12.015584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.015447 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3563a606-4dee-47cd-815f-ace2af65416b-kube-api-access-fdrpf" pod "3563a606-4dee-47cd-815f-ace2af65416b" (UID: "3563a606-4dee-47cd-815f-ace2af65416b"). InnerVolumeSpecName "kube-api-access-fdrpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:12.016731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.016702 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-istio-envoy" pod "28da3b98-e5f9-4355-876f-560f633061de" (UID: "28da3b98-e5f9-4355-876f-560f633061de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:12.036345 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.036078 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" Nov 03 20:26:12.105191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105156 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.105191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105189 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28da3b98-e5f9-4355-876f-560f633061de-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.105843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105206 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28da3b98-e5f9-4355-876f-560f633061de-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.105843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105219 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.105843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105231 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.105843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105244 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.105843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105256 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.105843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105268 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.105843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105282 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3563a606-4dee-47cd-815f-ace2af65416b-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.105843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105295 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.105843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105308 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3563a606-4dee-47cd-815f-ace2af65416b-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.105843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105322 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3563a606-4dee-47cd-815f-ace2af65416b-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.105843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105338 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fddjp\" (UniqueName: \"kubernetes.io/projected/28da3b98-e5f9-4355-876f-560f633061de-kube-api-access-fddjp\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.105843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105354 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da3b98-e5f9-4355-876f-560f633061de-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.105843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105367 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28da3b98-e5f9-4355-876f-560f633061de-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.105843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105380 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3563a606-4dee-47cd-815f-ace2af65416b-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.105843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.105394 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdrpf\" (UniqueName: \"kubernetes.io/projected/3563a606-4dee-47cd-815f-ace2af65416b-kube-api-access-fdrpf\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:12.108021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.107976 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" event={"ID":"873efceb-087b-4442-966e-e395b73ae3c5","Type":"ContainerDied","Data":"1dbe1721f86230c86482e04bd922ea2d09a39b99e62fc30cb1e9b9fb2cdb8f3a"} Nov 03 20:26:12.108151 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.107990 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz" Nov 03 20:26:12.108151 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.108135 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dbe1721f86230c86482e04bd922ea2d09a39b99e62fc30cb1e9b9fb2cdb8f3a" Nov 03 20:26:12.109483 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.109460 5111 generic.go:356] "Generic (PLEG): container finished" podID="3563a606-4dee-47cd-815f-ace2af65416b" containerID="5d9eef1047bf66d368a9a2e1b58a1a8a6b349ccbabb3e5a928fcae319695d96e" exitCode=0 Nov 03 20:26:12.109622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.109586 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" event={"ID":"3563a606-4dee-47cd-815f-ace2af65416b","Type":"ContainerDied","Data":"5d9eef1047bf66d368a9a2e1b58a1a8a6b349ccbabb3e5a928fcae319695d96e"} Nov 03 20:26:12.109713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.109630 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" event={"ID":"3563a606-4dee-47cd-815f-ace2af65416b","Type":"ContainerDied","Data":"5de821e627c2ff7c60a4d87df9cd3d0f83ba575d33c4125b6f623656f2c9936b"} Nov 03 20:26:12.109713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.109654 5111 scope.go:122] "RemoveContainer" containerID="5d9eef1047bf66d368a9a2e1b58a1a8a6b349ccbabb3e5a928fcae319695d96e" Nov 03 20:26:12.109826 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.109752 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s" Nov 03 20:26:12.113015 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.112990 5111 generic.go:356] "Generic (PLEG): container finished" podID="28da3b98-e5f9-4355-876f-560f633061de" containerID="308dfd08165678b1b04a95906e90078ecdbf6c87a114bfc8e4ebd0c4b608d59a" exitCode=0 Nov 03 20:26:12.113114 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.113038 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" event={"ID":"28da3b98-e5f9-4355-876f-560f633061de","Type":"ContainerDied","Data":"308dfd08165678b1b04a95906e90078ecdbf6c87a114bfc8e4ebd0c4b608d59a"} Nov 03 20:26:12.113114 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.113068 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" event={"ID":"28da3b98-e5f9-4355-876f-560f633061de","Type":"ContainerDied","Data":"a341df54eb7e9761e398160a15c3edb60595b88aba515a7f612ff12d2075e601"} Nov 03 20:26:12.113195 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.113127 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr" Nov 03 20:26:12.132081 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.132063 5111 scope.go:122] "RemoveContainer" containerID="5d9eef1047bf66d368a9a2e1b58a1a8a6b349ccbabb3e5a928fcae319695d96e" Nov 03 20:26:12.132484 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:12.132468 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9eef1047bf66d368a9a2e1b58a1a8a6b349ccbabb3e5a928fcae319695d96e\": container with ID starting with 5d9eef1047bf66d368a9a2e1b58a1a8a6b349ccbabb3e5a928fcae319695d96e not found: ID does not exist" containerID="5d9eef1047bf66d368a9a2e1b58a1a8a6b349ccbabb3e5a928fcae319695d96e" Nov 03 20:26:12.132861 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.132841 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9eef1047bf66d368a9a2e1b58a1a8a6b349ccbabb3e5a928fcae319695d96e"} err="failed to get container status \"5d9eef1047bf66d368a9a2e1b58a1a8a6b349ccbabb3e5a928fcae319695d96e\": rpc error: code = NotFound desc = could not find container \"5d9eef1047bf66d368a9a2e1b58a1a8a6b349ccbabb3e5a928fcae319695d96e\": container with ID starting with 5d9eef1047bf66d368a9a2e1b58a1a8a6b349ccbabb3e5a928fcae319695d96e not found: ID does not exist" Nov 03 20:26:12.132941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.132933 5111 scope.go:122] "RemoveContainer" containerID="308dfd08165678b1b04a95906e90078ecdbf6c87a114bfc8e4ebd0c4b608d59a" Nov 03 20:26:12.162363 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.162243 5111 scope.go:122] "RemoveContainer" containerID="308dfd08165678b1b04a95906e90078ecdbf6c87a114bfc8e4ebd0c4b608d59a" Nov 03 20:26:12.162722 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:12.162692 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"308dfd08165678b1b04a95906e90078ecdbf6c87a114bfc8e4ebd0c4b608d59a\": container with ID starting with 308dfd08165678b1b04a95906e90078ecdbf6c87a114bfc8e4ebd0c4b608d59a not found: ID does not exist" containerID="308dfd08165678b1b04a95906e90078ecdbf6c87a114bfc8e4ebd0c4b608d59a" Nov 03 20:26:12.162867 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.162733 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308dfd08165678b1b04a95906e90078ecdbf6c87a114bfc8e4ebd0c4b608d59a"} err="failed to get container status \"308dfd08165678b1b04a95906e90078ecdbf6c87a114bfc8e4ebd0c4b608d59a\": rpc error: code = NotFound desc = could not find container \"308dfd08165678b1b04a95906e90078ecdbf6c87a114bfc8e4ebd0c4b608d59a\": container with ID starting with 308dfd08165678b1b04a95906e90078ecdbf6c87a114bfc8e4ebd0c4b608d59a not found: ID does not exist" Nov 03 20:26:12.192956 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.192925 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s"] Nov 03 20:26:12.194391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.194369 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-47g7s"] Nov 03 20:26:12.194711 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.194697 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr"] Nov 03 20:26:12.199451 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.199420 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-l7szr"] Nov 03 20:26:12.248028 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.247980 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" podUID="af7b24bc-adce-4c7a-b2be-5844a4d44675" containerName="payload-pre-processing" containerID="cri-o://0002aa5a90ea141e32e45508480c90a1fb37a048cebf58148e9438179853dbb3" gracePeriod=25 Nov 03 20:26:12.362315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.362275 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z"] Nov 03 20:26:12.573804 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.573713 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf"] Nov 03 20:26:12.574627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.574521 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" podUID="d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25" containerName="maas-api" containerID="cri-o://0bbfc98f0956d085143594be6bd79313555cd2a6408094063391aace1a131767" gracePeriod=30 Nov 03 20:26:12.867378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.867345 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5"] Nov 03 20:26:12.871581 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:12.871551 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22a567-25bb5"] Nov 03 20:26:13.143938 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.143906 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" event={"ID":"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0","Type":"ContainerStarted","Data":"2e62c042c5e70ab2816162bf7e62d1affb98e13c9066cde3507437400c4d1e93"} Nov 03 20:26:13.144484 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.144455 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" event={"ID":"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0","Type":"ContainerStarted","Data":"1e51865b2900091b5ac897f25e617add99e1843cb6cc15beff8686747b41831a"} Nov 03 20:26:13.147096 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.147066 5111 generic.go:356] "Generic (PLEG): container finished" podID="d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25" containerID="0bbfc98f0956d085143594be6bd79313555cd2a6408094063391aace1a131767" exitCode=0 Nov 03 20:26:13.147224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.147166 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" event={"ID":"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25","Type":"ContainerDied","Data":"0bbfc98f0956d085143594be6bd79313555cd2a6408094063391aace1a131767"} Nov 03 20:26:13.170847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.170767 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" podStartSLOduration=2.170746645 podStartE2EDuration="2.170746645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:13.169937069 +0000 UTC m=+2803.132315508" watchObservedRunningTime="2026-11-03 20:26:13.170746645 +0000 UTC m=+2803.133125088" Nov 03 20:26:13.231254 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.231223 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:26:13.302983 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.302945 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb"] Nov 03 20:26:13.307571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.307420 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d81d18-676hb"] Nov 03 20:26:13.329544 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.329495 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-maas-api-tls\") pod \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\" (UID: \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\") " Nov 03 20:26:13.329768 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.329703 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-kube-api-access-gmmcv\" (UniqueName: \"kubernetes.io/projected/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-kube-api-access-gmmcv\") pod \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\" (UID: \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\") " Nov 03 20:26:13.329843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.329799 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-openshift-service-ca\") pod \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\" (UID: \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\") " Nov 03 20:26:13.329910 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.329863 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-metrics-certs\") pod \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\" (UID: \"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25\") " Nov 03 20:26:13.330314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.330286 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-openshift-service-ca" pod "d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25" (UID: "d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:13.330744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.330720 5111 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-openshift-service-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:13.332280 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.332251 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-maas-api-tls" pod "d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25" (UID: "d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:13.332685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.332657 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-kube-api-access-gmmcv" pod "d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25" (UID: "d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25"). InnerVolumeSpecName "kube-api-access-gmmcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:13.332930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.332906 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-metrics-certs" pod "d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25" (UID: "d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:13.432206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.432100 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-maas-api-tls\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:13.432206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.432132 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmmcv\" (UniqueName: \"kubernetes.io/projected/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-kube-api-access-gmmcv\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:13.432206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:13.432142 5111 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25-metrics-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:14.163759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.163725 5111 generic.go:356] "Generic (PLEG): container finished" podID="550ca9cd-5b2e-435d-a9d9-958f65b7e5a0" containerID="2e62c042c5e70ab2816162bf7e62d1affb98e13c9066cde3507437400c4d1e93" exitCode=0 Nov 03 20:26:14.164242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.163810 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" event={"ID":"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0","Type":"ContainerDied","Data":"2e62c042c5e70ab2816162bf7e62d1affb98e13c9066cde3507437400c4d1e93"} Nov 03 20:26:14.165744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.165721 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" Nov 03 20:26:14.165744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.165733 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf" event={"ID":"d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25","Type":"ContainerDied","Data":"507fb9b90169de8522fa04cb2feff6bc60ee49c0bb9f8edeee23d7ce2327f766"} Nov 03 20:26:14.165908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.165782 5111 scope.go:122] "RemoveContainer" containerID="0bbfc98f0956d085143594be6bd79313555cd2a6408094063391aace1a131767" Nov 03 20:26:14.188616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.188584 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28da3b98-e5f9-4355-876f-560f633061de" path="/var/lib/kubelet/pods/28da3b98-e5f9-4355-876f-560f633061de/volumes" Nov 03 20:26:14.189353 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.189279 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3563a606-4dee-47cd-815f-ace2af65416b" path="/var/lib/kubelet/pods/3563a606-4dee-47cd-815f-ace2af65416b/volumes" Nov 03 20:26:14.189898 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.189882 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe71f33-2cb2-437e-88b0-f544c9cbe663" path="/var/lib/kubelet/pods/afe71f33-2cb2-437e-88b0-f544c9cbe663/volumes" Nov 03 20:26:14.190898 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.190880 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61778c4-0880-4606-a5af-0ef731921ead" path="/var/lib/kubelet/pods/c61778c4-0880-4606-a5af-0ef731921ead/volumes" Nov 03 20:26:14.209078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.208841 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf"] Nov 03 20:26:14.216915 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.216884 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22a567-6dfc7d9857-pm2bf"] Nov 03 20:26:14.719462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.719437 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:26:14.751241 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.751208 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-podinfo\") pod \"97388cfd-a7f0-425d-946b-bf0181f548b0\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " Nov 03 20:26:14.751441 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.751292 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-envoy\") pod \"97388cfd-a7f0-425d-946b-bf0181f548b0\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " Nov 03 20:26:14.751441 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.751321 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-token\" (UniqueName: \"kubernetes.io/projected/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-token\") pod \"97388cfd-a7f0-425d-946b-bf0181f548b0\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " Nov 03 20:26:14.751561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.751498 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97388cfd-a7f0-425d-946b-bf0181f548b0-kube-api-access-5jm7s\" (UniqueName: \"kubernetes.io/projected/97388cfd-a7f0-425d-946b-bf0181f548b0-kube-api-access-5jm7s\") pod \"97388cfd-a7f0-425d-946b-bf0181f548b0\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " Nov 03 20:26:14.751630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.751609 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97388cfd-a7f0-425d-946b-bf0181f548b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97388cfd-a7f0-425d-946b-bf0181f548b0-istiod-ca-cert\") pod \"97388cfd-a7f0-425d-946b-bf0181f548b0\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " Nov 03 20:26:14.751689 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.751665 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-credential-socket\") pod \"97388cfd-a7f0-425d-946b-bf0181f548b0\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " Nov 03 20:26:14.751760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.751743 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-workload-socket\") pod \"97388cfd-a7f0-425d-946b-bf0181f548b0\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " Nov 03 20:26:14.751845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.751830 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-data\") pod \"97388cfd-a7f0-425d-946b-bf0181f548b0\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " Nov 03 20:26:14.751931 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.751913 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-workload-certs\") pod \"97388cfd-a7f0-425d-946b-bf0181f548b0\" (UID: \"97388cfd-a7f0-425d-946b-bf0181f548b0\") " Nov 03 20:26:14.752674 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.752644 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97388cfd-a7f0-425d-946b-bf0181f548b0-istiod-ca-cert" pod "97388cfd-a7f0-425d-946b-bf0181f548b0" (UID: "97388cfd-a7f0-425d-946b-bf0181f548b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:14.753312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.753288 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-credential-socket" pod "97388cfd-a7f0-425d-946b-bf0181f548b0" (UID: "97388cfd-a7f0-425d-946b-bf0181f548b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:14.753483 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.753308 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-data" pod "97388cfd-a7f0-425d-946b-bf0181f548b0" (UID: "97388cfd-a7f0-425d-946b-bf0181f548b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:14.753835 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.753788 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-workload-socket" pod "97388cfd-a7f0-425d-946b-bf0181f548b0" (UID: "97388cfd-a7f0-425d-946b-bf0181f548b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:14.754009 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.753987 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-workload-certs" pod "97388cfd-a7f0-425d-946b-bf0181f548b0" (UID: "97388cfd-a7f0-425d-946b-bf0181f548b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:14.754982 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.754952 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-podinfo" pod "97388cfd-a7f0-425d-946b-bf0181f548b0" (UID: "97388cfd-a7f0-425d-946b-bf0181f548b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:14.755105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.754968 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-token" pod "97388cfd-a7f0-425d-946b-bf0181f548b0" (UID: "97388cfd-a7f0-425d-946b-bf0181f548b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:14.762718 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.762658 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-envoy" pod "97388cfd-a7f0-425d-946b-bf0181f548b0" (UID: "97388cfd-a7f0-425d-946b-bf0181f548b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:14.763961 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.763931 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97388cfd-a7f0-425d-946b-bf0181f548b0-kube-api-access-5jm7s" pod "97388cfd-a7f0-425d-946b-bf0181f548b0" (UID: "97388cfd-a7f0-425d-946b-bf0181f548b0"). InnerVolumeSpecName "kube-api-access-5jm7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:14.855126 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.855068 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:14.855437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.855394 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:14.855629 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.855564 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jm7s\" (UniqueName: \"kubernetes.io/projected/97388cfd-a7f0-425d-946b-bf0181f548b0-kube-api-access-5jm7s\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:14.855629 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.855588 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97388cfd-a7f0-425d-946b-bf0181f548b0-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:14.855629 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.855602 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:14.856021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.855949 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:14.856021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.855983 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:14.856021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.855999 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97388cfd-a7f0-425d-946b-bf0181f548b0-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:14.856441 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:14.856384 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97388cfd-a7f0-425d-946b-bf0181f548b0-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:15.177990 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.177958 5111 generic.go:356] "Generic (PLEG): container finished" podID="97388cfd-a7f0-425d-946b-bf0181f548b0" containerID="49a572d4425500309fe26eac2e1d22038f7c103faee0d2fce8204dec65f313f5" exitCode=0 Nov 03 20:26:15.178477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.178008 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" Nov 03 20:26:15.178477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.178039 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" event={"ID":"97388cfd-a7f0-425d-946b-bf0181f548b0","Type":"ContainerDied","Data":"49a572d4425500309fe26eac2e1d22038f7c103faee0d2fce8204dec65f313f5"} Nov 03 20:26:15.178477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.178092 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx" event={"ID":"97388cfd-a7f0-425d-946b-bf0181f548b0","Type":"ContainerDied","Data":"c594298b69cb59dc05a52927e3f736e4e5c1d3c4a8f97241049049b443761b80"} Nov 03 20:26:15.178477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.178122 5111 scope.go:122] "RemoveContainer" containerID="49a572d4425500309fe26eac2e1d22038f7c103faee0d2fce8204dec65f313f5" Nov 03 20:26:15.197457 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.197428 5111 scope.go:122] "RemoveContainer" containerID="7066ba48898c82f9859b2cf5b411008139e2782c58d7ee14dc575085b7bcee1c" Nov 03 20:26:15.211961 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.211933 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx"] Nov 03 20:26:15.218820 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.218798 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-qcqlx"] Nov 03 20:26:15.222116 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.222092 5111 scope.go:122] "RemoveContainer" containerID="49a572d4425500309fe26eac2e1d22038f7c103faee0d2fce8204dec65f313f5" Nov 03 20:26:15.222601 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:15.222565 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a572d4425500309fe26eac2e1d22038f7c103faee0d2fce8204dec65f313f5\": container with ID starting with 49a572d4425500309fe26eac2e1d22038f7c103faee0d2fce8204dec65f313f5 not found: ID does not exist" containerID="49a572d4425500309fe26eac2e1d22038f7c103faee0d2fce8204dec65f313f5" Nov 03 20:26:15.222719 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.222628 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a572d4425500309fe26eac2e1d22038f7c103faee0d2fce8204dec65f313f5"} err="failed to get container status \"49a572d4425500309fe26eac2e1d22038f7c103faee0d2fce8204dec65f313f5\": rpc error: code = NotFound desc = could not find container \"49a572d4425500309fe26eac2e1d22038f7c103faee0d2fce8204dec65f313f5\": container with ID starting with 49a572d4425500309fe26eac2e1d22038f7c103faee0d2fce8204dec65f313f5 not found: ID does not exist" Nov 03 20:26:15.222719 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.222654 5111 scope.go:122] "RemoveContainer" containerID="7066ba48898c82f9859b2cf5b411008139e2782c58d7ee14dc575085b7bcee1c" Nov 03 20:26:15.223006 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:15.222989 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7066ba48898c82f9859b2cf5b411008139e2782c58d7ee14dc575085b7bcee1c\": container with ID starting with 7066ba48898c82f9859b2cf5b411008139e2782c58d7ee14dc575085b7bcee1c not found: ID does not exist" containerID="7066ba48898c82f9859b2cf5b411008139e2782c58d7ee14dc575085b7bcee1c" Nov 03 20:26:15.223107 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.223091 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7066ba48898c82f9859b2cf5b411008139e2782c58d7ee14dc575085b7bcee1c"} err="failed to get container status \"7066ba48898c82f9859b2cf5b411008139e2782c58d7ee14dc575085b7bcee1c\": rpc error: code = NotFound desc = could not find container \"7066ba48898c82f9859b2cf5b411008139e2782c58d7ee14dc575085b7bcee1c\": container with ID starting with 7066ba48898c82f9859b2cf5b411008139e2782c58d7ee14dc575085b7bcee1c not found: ID does not exist" Nov 03 20:26:15.343011 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.342982 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" Nov 03 20:26:15.404049 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.404020 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:26:15.468397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.468364 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-envoy\") pod \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " Nov 03 20:26:15.468599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.468412 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-podinfo\") pod \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " Nov 03 20:26:15.468599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.468446 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-data\") pod \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " Nov 03 20:26:15.468599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.468478 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istiod-ca-cert\") pod \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " Nov 03 20:26:15.468599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.468503 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-maas-api-openshift-ca\") pod \"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0\" (UID: \"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0\") " Nov 03 20:26:15.468599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.468558 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-credential-socket\") pod \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " Nov 03 20:26:15.468599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.468593 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-workload-certs\") pod \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " Nov 03 20:26:15.468973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.468648 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-maas-api-xks-ca\") pod \"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0\" (UID: \"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0\") " Nov 03 20:26:15.468973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.468695 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-token\" (UniqueName: \"kubernetes.io/projected/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-token\") pod \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " Nov 03 20:26:15.468973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.468747 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-workload-socket\") pod \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " Nov 03 20:26:15.468973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.468805 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8edc2022-355d-4c8e-9564-f2c0a2c6b397-kube-api-access-2dntc\" (UniqueName: \"kubernetes.io/projected/8edc2022-355d-4c8e-9564-f2c0a2c6b397-kube-api-access-2dntc\") pod \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\" (UID: \"8edc2022-355d-4c8e-9564-f2c0a2c6b397\") " Nov 03 20:26:15.468973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.468893 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-kube-api-access-qdmxb\" (UniqueName: \"kubernetes.io/projected/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-kube-api-access-qdmxb\") pod \"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0\" (UID: \"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0\") " Nov 03 20:26:15.472601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.472527 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-token" pod "8edc2022-355d-4c8e-9564-f2c0a2c6b397" (UID: "8edc2022-355d-4c8e-9564-f2c0a2c6b397"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:15.472871 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.472675 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-data" pod "8edc2022-355d-4c8e-9564-f2c0a2c6b397" (UID: "8edc2022-355d-4c8e-9564-f2c0a2c6b397"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:15.472976 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.472932 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-podinfo" pod "8edc2022-355d-4c8e-9564-f2c0a2c6b397" (UID: "8edc2022-355d-4c8e-9564-f2c0a2c6b397"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:15.473041 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.472910 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-envoy" pod "8edc2022-355d-4c8e-9564-f2c0a2c6b397" (UID: "8edc2022-355d-4c8e-9564-f2c0a2c6b397"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:15.473226 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.473193 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istiod-ca-cert" pod "8edc2022-355d-4c8e-9564-f2c0a2c6b397" (UID: "8edc2022-355d-4c8e-9564-f2c0a2c6b397"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:15.474734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.473681 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-workload-certs" pod "8edc2022-355d-4c8e-9564-f2c0a2c6b397" (UID: "8edc2022-355d-4c8e-9564-f2c0a2c6b397"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:15.474734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.473956 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-maas-api-openshift-ca" pod "550ca9cd-5b2e-435d-a9d9-958f65b7e5a0" (UID: "550ca9cd-5b2e-435d-a9d9-958f65b7e5a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:15.474734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.473954 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-credential-socket" pod "8edc2022-355d-4c8e-9564-f2c0a2c6b397" (UID: "8edc2022-355d-4c8e-9564-f2c0a2c6b397"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:15.474734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.474003 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-workload-socket" pod "8edc2022-355d-4c8e-9564-f2c0a2c6b397" (UID: "8edc2022-355d-4c8e-9564-f2c0a2c6b397"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:15.474734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.474675 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-kube-api-access-qdmxb" pod "550ca9cd-5b2e-435d-a9d9-958f65b7e5a0" (UID: "550ca9cd-5b2e-435d-a9d9-958f65b7e5a0"). InnerVolumeSpecName "kube-api-access-qdmxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:15.476672 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.476636 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8edc2022-355d-4c8e-9564-f2c0a2c6b397-kube-api-access-2dntc" pod "8edc2022-355d-4c8e-9564-f2c0a2c6b397" (UID: "8edc2022-355d-4c8e-9564-f2c0a2c6b397"). InnerVolumeSpecName "kube-api-access-2dntc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:15.477041 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.477017 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-maas-api-xks-ca" pod "550ca9cd-5b2e-435d-a9d9-958f65b7e5a0" (UID: "550ca9cd-5b2e-435d-a9d9-958f65b7e5a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:15.570511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.570473 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:15.570511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.570504 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:15.570511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.570513 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:15.570793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.570524 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dntc\" (UniqueName: \"kubernetes.io/projected/8edc2022-355d-4c8e-9564-f2c0a2c6b397-kube-api-access-2dntc\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:15.570793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.570566 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdmxb\" (UniqueName: \"kubernetes.io/projected/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-kube-api-access-qdmxb\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:15.570793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.570581 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:15.570793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.570590 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:15.570793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.570598 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:15.570793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.570626 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edc2022-355d-4c8e-9564-f2c0a2c6b397-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:15.570793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.570635 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:15.570793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.570643 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:15.570793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:15.570652 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edc2022-355d-4c8e-9564-f2c0a2c6b397-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:16.138047 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.138009 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:26:16.182078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.182040 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-podinfo\") pod \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " Nov 03 20:26:16.182640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.182122 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-kube-api-access-vh5jk\" (UniqueName: \"kubernetes.io/projected/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-kube-api-access-vh5jk\") pod \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " Nov 03 20:26:16.182640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.182161 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istiod-ca-cert\") pod \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " Nov 03 20:26:16.182640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.182227 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-envoy\") pod \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " Nov 03 20:26:16.182640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.182315 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-workload-socket\") pod \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " Nov 03 20:26:16.182640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.182335 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-data\") pod \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " Nov 03 20:26:16.182640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.182383 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-workload-certs\") pod \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " Nov 03 20:26:16.182640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.182414 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-token\" (UniqueName: \"kubernetes.io/projected/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-token\") pod \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " Nov 03 20:26:16.182640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.182509 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-credential-socket\") pod \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\" (UID: \"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2\") " Nov 03 20:26:16.183390 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.183363 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-credential-socket" pod "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" (UID: "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:16.183575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.183553 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-workload-socket" pod "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" (UID: "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:16.183766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.183745 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-data" pod "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" (UID: "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:16.183988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.183949 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-workload-certs" pod "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" (UID: "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:16.187467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.187446 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-token" pod "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" (UID: "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:16.192440 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.192079 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-kube-api-access-vh5jk" pod "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" (UID: "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2"). InnerVolumeSpecName "kube-api-access-vh5jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:16.193868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.193029 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istiod-ca-cert" pod "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" (UID: "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:16.195326 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.195168 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" Nov 03 20:26:16.198511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.197682 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-podinfo" pod "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" (UID: "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:16.205027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.205000 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97388cfd-a7f0-425d-946b-bf0181f548b0" path="/var/lib/kubelet/pods/97388cfd-a7f0-425d-946b-bf0181f548b0/volumes" Nov 03 20:26:16.205807 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.205783 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-envoy" pod "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" (UID: "1215d5ec-fd6a-40cb-a43c-9a51d68d70f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:16.205964 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.205868 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25" path="/var/lib/kubelet/pods/d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25/volumes" Nov 03 20:26:16.207586 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.207559 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z" event={"ID":"550ca9cd-5b2e-435d-a9d9-958f65b7e5a0","Type":"ContainerDied","Data":"1e51865b2900091b5ac897f25e617add99e1843cb6cc15beff8686747b41831a"} Nov 03 20:26:16.207751 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.207725 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e51865b2900091b5ac897f25e617add99e1843cb6cc15beff8686747b41831a" Nov 03 20:26:16.211161 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.211134 5111 generic.go:356] "Generic (PLEG): container finished" podID="1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" containerID="54d095a0b7b62c3ae44ba288a768ae9a5ec315a2df862692323fa8e00c2a468e" exitCode=0 Nov 03 20:26:16.212428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.211446 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" Nov 03 20:26:16.212428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.211490 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" event={"ID":"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2","Type":"ContainerDied","Data":"54d095a0b7b62c3ae44ba288a768ae9a5ec315a2df862692323fa8e00c2a468e"} Nov 03 20:26:16.212428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.211552 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v" event={"ID":"1215d5ec-fd6a-40cb-a43c-9a51d68d70f2","Type":"ContainerDied","Data":"9a930ef38a716ad71f0c54a45e45d81fe19971a9f82b0e30afd56c6f18289465"} Nov 03 20:26:16.212428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.211583 5111 scope.go:122] "RemoveContainer" containerID="54d095a0b7b62c3ae44ba288a768ae9a5ec315a2df862692323fa8e00c2a468e" Nov 03 20:26:16.216660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.214634 5111 generic.go:356] "Generic (PLEG): container finished" podID="8edc2022-355d-4c8e-9564-f2c0a2c6b397" containerID="6bca2a51b6953dbbd9c11fd32777b34f6b955e3ff0dd1b6761d2dd5a21d4b555" exitCode=0 Nov 03 20:26:16.216660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.214685 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" Nov 03 20:26:16.216660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.214712 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" event={"ID":"8edc2022-355d-4c8e-9564-f2c0a2c6b397","Type":"ContainerDied","Data":"6bca2a51b6953dbbd9c11fd32777b34f6b955e3ff0dd1b6761d2dd5a21d4b555"} Nov 03 20:26:16.216660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.214754 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj" event={"ID":"8edc2022-355d-4c8e-9564-f2c0a2c6b397","Type":"ContainerDied","Data":"f3756c1cd5e3971427c1c8937f9a9cf1dc0a8d6594721a2869959919612eda0a"} Nov 03 20:26:16.237605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.237586 5111 scope.go:122] "RemoveContainer" containerID="630154985d3ce0bb2176b312942bff177fd8d8a62d52355211ee9750e94bbefe" Nov 03 20:26:16.253113 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.253091 5111 scope.go:122] "RemoveContainer" containerID="54d095a0b7b62c3ae44ba288a768ae9a5ec315a2df862692323fa8e00c2a468e" Nov 03 20:26:16.253631 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:16.253603 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d095a0b7b62c3ae44ba288a768ae9a5ec315a2df862692323fa8e00c2a468e\": container with ID starting with 54d095a0b7b62c3ae44ba288a768ae9a5ec315a2df862692323fa8e00c2a468e not found: ID does not exist" containerID="54d095a0b7b62c3ae44ba288a768ae9a5ec315a2df862692323fa8e00c2a468e" Nov 03 20:26:16.253714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.253643 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d095a0b7b62c3ae44ba288a768ae9a5ec315a2df862692323fa8e00c2a468e"} err="failed to get container status \"54d095a0b7b62c3ae44ba288a768ae9a5ec315a2df862692323fa8e00c2a468e\": rpc error: code = NotFound desc = could not find container \"54d095a0b7b62c3ae44ba288a768ae9a5ec315a2df862692323fa8e00c2a468e\": container with ID starting with 54d095a0b7b62c3ae44ba288a768ae9a5ec315a2df862692323fa8e00c2a468e not found: ID does not exist" Nov 03 20:26:16.253714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.253668 5111 scope.go:122] "RemoveContainer" containerID="630154985d3ce0bb2176b312942bff177fd8d8a62d52355211ee9750e94bbefe" Nov 03 20:26:16.253944 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:16.253923 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"630154985d3ce0bb2176b312942bff177fd8d8a62d52355211ee9750e94bbefe\": container with ID starting with 630154985d3ce0bb2176b312942bff177fd8d8a62d52355211ee9750e94bbefe not found: ID does not exist" containerID="630154985d3ce0bb2176b312942bff177fd8d8a62d52355211ee9750e94bbefe" Nov 03 20:26:16.254017 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.253953 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630154985d3ce0bb2176b312942bff177fd8d8a62d52355211ee9750e94bbefe"} err="failed to get container status \"630154985d3ce0bb2176b312942bff177fd8d8a62d52355211ee9750e94bbefe\": rpc error: code = NotFound desc = could not find container \"630154985d3ce0bb2176b312942bff177fd8d8a62d52355211ee9750e94bbefe\": container with ID starting with 630154985d3ce0bb2176b312942bff177fd8d8a62d52355211ee9750e94bbefe not found: ID does not exist" Nov 03 20:26:16.254017 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.253971 5111 scope.go:122] "RemoveContainer" containerID="6bca2a51b6953dbbd9c11fd32777b34f6b955e3ff0dd1b6761d2dd5a21d4b555" Nov 03 20:26:16.257729 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.257704 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj"] Nov 03 20:26:16.261558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.261525 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-66mlj"] Nov 03 20:26:16.266705 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.266683 5111 scope.go:122] "RemoveContainer" containerID="6bca2a51b6953dbbd9c11fd32777b34f6b955e3ff0dd1b6761d2dd5a21d4b555" Nov 03 20:26:16.266995 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:16.266975 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bca2a51b6953dbbd9c11fd32777b34f6b955e3ff0dd1b6761d2dd5a21d4b555\": container with ID starting with 6bca2a51b6953dbbd9c11fd32777b34f6b955e3ff0dd1b6761d2dd5a21d4b555 not found: ID does not exist" containerID="6bca2a51b6953dbbd9c11fd32777b34f6b955e3ff0dd1b6761d2dd5a21d4b555" Nov 03 20:26:16.267047 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.267003 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bca2a51b6953dbbd9c11fd32777b34f6b955e3ff0dd1b6761d2dd5a21d4b555"} err="failed to get container status \"6bca2a51b6953dbbd9c11fd32777b34f6b955e3ff0dd1b6761d2dd5a21d4b555\": rpc error: code = NotFound desc = could not find container \"6bca2a51b6953dbbd9c11fd32777b34f6b955e3ff0dd1b6761d2dd5a21d4b555\": container with ID starting with 6bca2a51b6953dbbd9c11fd32777b34f6b955e3ff0dd1b6761d2dd5a21d4b555 not found: ID does not exist" Nov 03 20:26:16.267100 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.267087 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v"] Nov 03 20:26:16.271035 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.271013 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-9ft8v"] Nov 03 20:26:16.285224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.285203 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:16.285224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.285225 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:16.285374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.285238 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:16.285374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.285258 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:16.285374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.285268 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:16.285374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.285277 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:16.285374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.285287 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:16.285374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.285295 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh5jk\" (UniqueName: \"kubernetes.io/projected/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-kube-api-access-vh5jk\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:16.285374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.285304 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:16.299128 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:16.299100 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg"] Nov 03 20:26:17.128363 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.128330 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp"] Nov 03 20:26:17.324398 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.324366 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr"] Nov 03 20:26:17.325209 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325183 5111 cpu_manager.go:414] "containerMap: removing container" podUID="28da3b98-e5f9-4355-876f-560f633061de" containerName="istio-proxy" Nov 03 20:26:17.325209 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325205 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28da3b98-e5f9-4355-876f-560f633061de" containerName="istio-proxy" Nov 03 20:26:17.325351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325219 5111 cpu_manager.go:414] "containerMap: removing container" podUID="d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25" containerName="maas-api" Nov 03 20:26:17.325351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325224 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25" containerName="maas-api" Nov 03 20:26:17.325351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325242 5111 cpu_manager.go:414] "containerMap: removing container" podUID="1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" containerName="istio-proxy" Nov 03 20:26:17.325351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325248 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" containerName="istio-proxy" Nov 03 20:26:17.325351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325265 5111 cpu_manager.go:414] "containerMap: removing container" podUID="97388cfd-a7f0-425d-946b-bf0181f548b0" containerName="istio-proxy" Nov 03 20:26:17.325351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325270 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97388cfd-a7f0-425d-946b-bf0181f548b0" containerName="istio-proxy" Nov 03 20:26:17.325351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325282 5111 cpu_manager.go:414] "containerMap: removing container" podUID="550ca9cd-5b2e-435d-a9d9-958f65b7e5a0" containerName="revoke-keys" Nov 03 20:26:17.325351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325286 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550ca9cd-5b2e-435d-a9d9-958f65b7e5a0" containerName="revoke-keys" Nov 03 20:26:17.325351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325306 5111 cpu_manager.go:414] "containerMap: removing container" podUID="8edc2022-355d-4c8e-9564-f2c0a2c6b397" containerName="istio-proxy" Nov 03 20:26:17.325351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325310 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8edc2022-355d-4c8e-9564-f2c0a2c6b397" containerName="istio-proxy" Nov 03 20:26:17.325351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325324 5111 cpu_manager.go:414] "containerMap: removing container" podUID="3563a606-4dee-47cd-815f-ace2af65416b" containerName="istio-proxy" Nov 03 20:26:17.325351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325329 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3563a606-4dee-47cd-815f-ace2af65416b" containerName="istio-proxy" Nov 03 20:26:17.325753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325424 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="97388cfd-a7f0-425d-946b-bf0181f548b0" containerName="istio-proxy" Nov 03 20:26:17.325753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325433 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="97388cfd-a7f0-425d-946b-bf0181f548b0" containerName="istio-proxy" Nov 03 20:26:17.325753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325439 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="550ca9cd-5b2e-435d-a9d9-958f65b7e5a0" containerName="revoke-keys" Nov 03 20:26:17.325753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325448 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="8edc2022-355d-4c8e-9564-f2c0a2c6b397" containerName="istio-proxy" Nov 03 20:26:17.325753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325455 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="3563a606-4dee-47cd-815f-ace2af65416b" containerName="istio-proxy" Nov 03 20:26:17.325753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325461 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" containerName="istio-proxy" Nov 03 20:26:17.325753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325469 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="28da3b98-e5f9-4355-876f-560f633061de" containerName="istio-proxy" Nov 03 20:26:17.325753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325476 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9d3a0d1-0fd0-4c23-b448-bda0ceb3cd25" containerName="maas-api" Nov 03 20:26:17.325753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.325482 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" containerName="istio-proxy" Nov 03 20:26:17.333094 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.333060 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr" Nov 03 20:26:17.338284 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.338256 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:26:17.338449 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.338359 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:26:17.338513 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.338465 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:26:17.340430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.340124 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr"] Nov 03 20:26:17.343339 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.343317 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:26:17.410781 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.402773 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b041b8d-92e6-488e-809c-40cfa2758f3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7907172-fhcqr\" (UID: \"0b041b8d-92e6-488e-809c-40cfa2758f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr" Nov 03 20:26:17.499885 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.499849 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rd75\" (UniqueName: \"kubernetes.io/projected/0b041b8d-92e6-488e-809c-40cfa2758f3e-kube-api-access-7rd75\") pod \"maas-api-revoke-sub-d7907172-fhcqr\" (UID: \"0b041b8d-92e6-488e-809c-40cfa2758f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr" Nov 03 20:26:17.534039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.533991 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b041b8d-92e6-488e-809c-40cfa2758f3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7907172-fhcqr\" (UID: \"0b041b8d-92e6-488e-809c-40cfa2758f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr" Nov 03 20:26:17.635795 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.635751 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b041b8d-92e6-488e-809c-40cfa2758f3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7907172-fhcqr\" (UID: \"0b041b8d-92e6-488e-809c-40cfa2758f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr" Nov 03 20:26:17.636042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.635916 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rd75\" (UniqueName: \"kubernetes.io/projected/0b041b8d-92e6-488e-809c-40cfa2758f3e-kube-api-access-7rd75\") pod \"maas-api-revoke-sub-d7907172-fhcqr\" (UID: \"0b041b8d-92e6-488e-809c-40cfa2758f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr" Nov 03 20:26:17.636042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.635950 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b041b8d-92e6-488e-809c-40cfa2758f3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7907172-fhcqr\" (UID: \"0b041b8d-92e6-488e-809c-40cfa2758f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr" Nov 03 20:26:17.636734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.636693 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b041b8d-92e6-488e-809c-40cfa2758f3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7907172-fhcqr\" (UID: \"0b041b8d-92e6-488e-809c-40cfa2758f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr" Nov 03 20:26:17.638519 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.638491 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b041b8d-92e6-488e-809c-40cfa2758f3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7907172-fhcqr\" (UID: \"0b041b8d-92e6-488e-809c-40cfa2758f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr" Nov 03 20:26:17.644022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.643994 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rd75\" (UniqueName: \"kubernetes.io/projected/0b041b8d-92e6-488e-809c-40cfa2758f3e-kube-api-access-7rd75\") pod \"maas-api-revoke-sub-d7907172-fhcqr\" (UID: \"0b041b8d-92e6-488e-809c-40cfa2758f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr" Nov 03 20:26:17.656845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.656817 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr" Nov 03 20:26:17.943873 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:17.943830 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr"] Nov 03 20:26:17.944643 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:26:17.944613 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b041b8d_92e6_488e_809c_40cfa2758f3e.slice/crio-078b3c4b6761fe90ae2a6169aad875b6419757372180406883215af197bc2b72 WatchSource:0}: Error finding container 078b3c4b6761fe90ae2a6169aad875b6419757372180406883215af197bc2b72: Status 404 returned error can't find the container with id 078b3c4b6761fe90ae2a6169aad875b6419757372180406883215af197bc2b72 Nov 03 20:26:18.184900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:18.184868 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" path="/var/lib/kubelet/pods/1215d5ec-fd6a-40cb-a43c-9a51d68d70f2/volumes" Nov 03 20:26:18.185486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:18.185462 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8edc2022-355d-4c8e-9564-f2c0a2c6b397" path="/var/lib/kubelet/pods/8edc2022-355d-4c8e-9564-f2c0a2c6b397/volumes" Nov 03 20:26:18.248479 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:18.248390 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr" event={"ID":"0b041b8d-92e6-488e-809c-40cfa2758f3e","Type":"ContainerStarted","Data":"078b3c4b6761fe90ae2a6169aad875b6419757372180406883215af197bc2b72"} Nov 03 20:26:19.260135 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:19.260095 5111 generic.go:356] "Generic (PLEG): container finished" podID="0b041b8d-92e6-488e-809c-40cfa2758f3e" containerID="641e17b3fbb901df9bef5c82e6d06aa6e1cd57a287876a843717da88f93e6a3f" exitCode=0 Nov 03 20:26:19.260596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:19.260187 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr" event={"ID":"0b041b8d-92e6-488e-809c-40cfa2758f3e","Type":"ContainerDied","Data":"641e17b3fbb901df9bef5c82e6d06aa6e1cd57a287876a843717da88f93e6a3f"} Nov 03 20:26:20.511215 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.511186 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr" Nov 03 20:26:20.612385 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.612301 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx"] Nov 03 20:26:20.612987 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.612971 5111 cpu_manager.go:414] "containerMap: removing container" podUID="1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" containerName="istio-proxy" Nov 03 20:26:20.613030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.612989 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1215d5ec-fd6a-40cb-a43c-9a51d68d70f2" containerName="istio-proxy" Nov 03 20:26:20.613030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.613019 5111 cpu_manager.go:414] "containerMap: removing container" podUID="97388cfd-a7f0-425d-946b-bf0181f548b0" containerName="istio-proxy" Nov 03 20:26:20.613030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.613025 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97388cfd-a7f0-425d-946b-bf0181f548b0" containerName="istio-proxy" Nov 03 20:26:20.613155 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.613059 5111 cpu_manager.go:414] "containerMap: removing container" podUID="0b041b8d-92e6-488e-809c-40cfa2758f3e" containerName="revoke-keys" Nov 03 20:26:20.613155 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.613064 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b041b8d-92e6-488e-809c-40cfa2758f3e" containerName="revoke-keys" Nov 03 20:26:20.613241 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.613173 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b041b8d-92e6-488e-809c-40cfa2758f3e" containerName="revoke-keys" Nov 03 20:26:20.615634 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.615601 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b041b8d-92e6-488e-809c-40cfa2758f3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b041b8d-92e6-488e-809c-40cfa2758f3e-maas-api-openshift-ca\") pod \"0b041b8d-92e6-488e-809c-40cfa2758f3e\" (UID: \"0b041b8d-92e6-488e-809c-40cfa2758f3e\") " Nov 03 20:26:20.615763 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.615656 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b041b8d-92e6-488e-809c-40cfa2758f3e-kube-api-access-7rd75\" (UniqueName: \"kubernetes.io/projected/0b041b8d-92e6-488e-809c-40cfa2758f3e-kube-api-access-7rd75\") pod \"0b041b8d-92e6-488e-809c-40cfa2758f3e\" (UID: \"0b041b8d-92e6-488e-809c-40cfa2758f3e\") " Nov 03 20:26:20.615864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.615848 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b041b8d-92e6-488e-809c-40cfa2758f3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b041b8d-92e6-488e-809c-40cfa2758f3e-maas-api-xks-ca\") pod \"0b041b8d-92e6-488e-809c-40cfa2758f3e\" (UID: \"0b041b8d-92e6-488e-809c-40cfa2758f3e\") " Nov 03 20:26:20.616001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.615975 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b041b8d-92e6-488e-809c-40cfa2758f3e-maas-api-openshift-ca" pod "0b041b8d-92e6-488e-809c-40cfa2758f3e" (UID: "0b041b8d-92e6-488e-809c-40cfa2758f3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:20.616487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.616463 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b041b8d-92e6-488e-809c-40cfa2758f3e-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:20.617871 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.617849 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b041b8d-92e6-488e-809c-40cfa2758f3e-kube-api-access-7rd75" pod "0b041b8d-92e6-488e-809c-40cfa2758f3e" (UID: "0b041b8d-92e6-488e-809c-40cfa2758f3e"). InnerVolumeSpecName "kube-api-access-7rd75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:20.617991 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.617913 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b041b8d-92e6-488e-809c-40cfa2758f3e-maas-api-xks-ca" pod "0b041b8d-92e6-488e-809c-40cfa2758f3e" (UID: "0b041b8d-92e6-488e-809c-40cfa2758f3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:20.621147 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.621125 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" Nov 03 20:26:20.633514 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.633484 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx"] Nov 03 20:26:20.726281 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.726218 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlr5b\" (UniqueName: \"kubernetes.io/projected/384309b2-620c-49b2-818f-793652ba6cc6-kube-api-access-mlr5b\") pod \"maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx\" (UID: \"384309b2-620c-49b2-818f-793652ba6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" Nov 03 20:26:20.726489 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.726302 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/384309b2-620c-49b2-818f-793652ba6cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx\" (UID: \"384309b2-620c-49b2-818f-793652ba6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" Nov 03 20:26:20.726489 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.726368 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/384309b2-620c-49b2-818f-793652ba6cc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx\" (UID: \"384309b2-620c-49b2-818f-793652ba6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" Nov 03 20:26:20.726674 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.726653 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b041b8d-92e6-488e-809c-40cfa2758f3e-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:20.726737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.726683 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rd75\" (UniqueName: \"kubernetes.io/projected/0b041b8d-92e6-488e-809c-40cfa2758f3e-kube-api-access-7rd75\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:20.827663 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.827629 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlr5b\" (UniqueName: \"kubernetes.io/projected/384309b2-620c-49b2-818f-793652ba6cc6-kube-api-access-mlr5b\") pod \"maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx\" (UID: \"384309b2-620c-49b2-818f-793652ba6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" Nov 03 20:26:20.827889 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.827696 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/384309b2-620c-49b2-818f-793652ba6cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx\" (UID: \"384309b2-620c-49b2-818f-793652ba6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" Nov 03 20:26:20.827889 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.827773 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/384309b2-620c-49b2-818f-793652ba6cc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx\" (UID: \"384309b2-620c-49b2-818f-793652ba6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" Nov 03 20:26:20.828437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.828413 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/384309b2-620c-49b2-818f-793652ba6cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx\" (UID: \"384309b2-620c-49b2-818f-793652ba6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" Nov 03 20:26:20.830339 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.830316 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/384309b2-620c-49b2-818f-793652ba6cc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx\" (UID: \"384309b2-620c-49b2-818f-793652ba6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" Nov 03 20:26:20.835567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.835512 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlr5b\" (UniqueName: \"kubernetes.io/projected/384309b2-620c-49b2-818f-793652ba6cc6-kube-api-access-mlr5b\") pod \"maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx\" (UID: \"384309b2-620c-49b2-818f-793652ba6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" Nov 03 20:26:20.982834 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:20.982798 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" Nov 03 20:26:21.331724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.331646 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr" Nov 03 20:26:21.331724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.331666 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr" event={"ID":"0b041b8d-92e6-488e-809c-40cfa2758f3e","Type":"ContainerDied","Data":"078b3c4b6761fe90ae2a6169aad875b6419757372180406883215af197bc2b72"} Nov 03 20:26:21.331724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.331712 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="078b3c4b6761fe90ae2a6169aad875b6419757372180406883215af197bc2b72" Nov 03 20:26:21.436194 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.436158 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk"] Nov 03 20:26:21.444311 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.444276 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk" Nov 03 20:26:21.447424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.447383 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwdhl\" (UniqueName: \"kubernetes.io/projected/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-kube-api-access-jwdhl\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk\" (UID: \"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk" Nov 03 20:26:21.447595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.447484 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk\" (UID: \"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk" Nov 03 20:26:21.447595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.447578 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk\" (UID: \"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk" Nov 03 20:26:21.463159 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.462778 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx"] Nov 03 20:26:21.463939 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:26:21.463906 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384309b2_620c_49b2_818f_793652ba6cc6.slice/crio-e2f570950314d68ad6478fe545fd5e5b8b74c330f962b4350846146a43272abb WatchSource:0}: Error finding container e2f570950314d68ad6478fe545fd5e5b8b74c330f962b4350846146a43272abb: Status 404 returned error can't find the container with id e2f570950314d68ad6478fe545fd5e5b8b74c330f962b4350846146a43272abb Nov 03 20:26:21.467801 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.467776 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk"] Nov 03 20:26:21.548936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.548907 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk\" (UID: \"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk" Nov 03 20:26:21.549348 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.548969 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk\" (UID: \"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk" Nov 03 20:26:21.549348 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.549114 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwdhl\" (UniqueName: \"kubernetes.io/projected/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-kube-api-access-jwdhl\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk\" (UID: \"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk" Nov 03 20:26:21.549662 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.549642 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk\" (UID: \"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk" Nov 03 20:26:21.551487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.551464 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk\" (UID: \"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk" Nov 03 20:26:21.594062 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.593987 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwdhl\" (UniqueName: \"kubernetes.io/projected/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-kube-api-access-jwdhl\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk\" (UID: \"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk" Nov 03 20:26:21.749549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.749496 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5"] Nov 03 20:26:21.773820 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:21.773785 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk" Nov 03 20:26:22.126641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:22.126593 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk"] Nov 03 20:26:22.140301 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:22.130620 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" podUID="c24d736c-014e-45e0-9459-bd36e2556f15" containerName="payload-pre-processing" containerID="cri-o://199848a9c28a5e7c178afcb1634023e85dd9335367d3adafe46cc58b27ed605c" gracePeriod=25 Nov 03 20:26:22.144671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:22.144589 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" podUID="af7b24bc-adce-4c7a-b2be-5844a4d44675" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:26:22.353891 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:22.353849 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" event={"ID":"384309b2-620c-49b2-818f-793652ba6cc6","Type":"ContainerStarted","Data":"d373c09ca3881198a3a587aadc6b9da7994bc4a22ff41141fc56b770f127c62a"} Nov 03 20:26:22.354084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:22.353918 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" event={"ID":"384309b2-620c-49b2-818f-793652ba6cc6","Type":"ContainerStarted","Data":"e2f570950314d68ad6478fe545fd5e5b8b74c330f962b4350846146a43272abb"} Nov 03 20:26:22.356814 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:22.356780 5111 generic.go:356] "Generic (PLEG): container finished" podID="c24d736c-014e-45e0-9459-bd36e2556f15" containerID="199848a9c28a5e7c178afcb1634023e85dd9335367d3adafe46cc58b27ed605c" exitCode=0 Nov 03 20:26:22.356956 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:22.356816 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" event={"ID":"c24d736c-014e-45e0-9459-bd36e2556f15","Type":"ContainerDied","Data":"199848a9c28a5e7c178afcb1634023e85dd9335367d3adafe46cc58b27ed605c"} Nov 03 20:26:22.358735 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:22.358706 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk" event={"ID":"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc","Type":"ContainerStarted","Data":"2357f9e4de118ef675b6a242a0b94a91e746eff13e053cf76dbd641419d69986"} Nov 03 20:26:22.380633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:22.378128 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" podStartSLOduration=2.378107982 podStartE2EDuration="2.378107982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:22.372301326 +0000 UTC m=+2812.334679765" watchObservedRunningTime="2026-11-03 20:26:22.378107982 +0000 UTC m=+2812.340486420" Nov 03 20:26:22.474312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:22.474281 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" Nov 03 20:26:22.568832 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:22.568794 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24d736c-014e-45e0-9459-bd36e2556f15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24d736c-014e-45e0-9459-bd36e2556f15-plugins-config-volume\") pod \"c24d736c-014e-45e0-9459-bd36e2556f15\" (UID: \"c24d736c-014e-45e0-9459-bd36e2556f15\") " Nov 03 20:26:22.569213 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:22.568900 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24d736c-014e-45e0-9459-bd36e2556f15-kube-api-access-gjdcf\" (UniqueName: \"kubernetes.io/projected/c24d736c-014e-45e0-9459-bd36e2556f15-kube-api-access-gjdcf\") pod \"c24d736c-014e-45e0-9459-bd36e2556f15\" (UID: \"c24d736c-014e-45e0-9459-bd36e2556f15\") " Nov 03 20:26:22.569213 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:22.569169 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24d736c-014e-45e0-9459-bd36e2556f15-plugins-config-volume" pod "c24d736c-014e-45e0-9459-bd36e2556f15" (UID: "c24d736c-014e-45e0-9459-bd36e2556f15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:22.569526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:22.569506 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24d736c-014e-45e0-9459-bd36e2556f15-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:22.645647 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:22.645558 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24d736c-014e-45e0-9459-bd36e2556f15-kube-api-access-gjdcf" pod "c24d736c-014e-45e0-9459-bd36e2556f15" (UID: "c24d736c-014e-45e0-9459-bd36e2556f15"). InnerVolumeSpecName "kube-api-access-gjdcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:22.679785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:22.679727 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjdcf\" (UniqueName: \"kubernetes.io/projected/c24d736c-014e-45e0-9459-bd36e2556f15-kube-api-access-gjdcf\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.341290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:23.341251 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m"] Nov 03 20:26:23.353942 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:23.353913 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a476f8-jmj9m"] Nov 03 20:26:23.370470 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:23.370444 5111 generic.go:356] "Generic (PLEG): container finished" podID="384309b2-620c-49b2-818f-793652ba6cc6" containerID="d373c09ca3881198a3a587aadc6b9da7994bc4a22ff41141fc56b770f127c62a" exitCode=0 Nov 03 20:26:23.370652 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:23.370553 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" event={"ID":"384309b2-620c-49b2-818f-793652ba6cc6","Type":"ContainerDied","Data":"d373c09ca3881198a3a587aadc6b9da7994bc4a22ff41141fc56b770f127c62a"} Nov 03 20:26:23.372510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:23.372483 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" Nov 03 20:26:23.372665 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:23.372487 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp" event={"ID":"c24d736c-014e-45e0-9459-bd36e2556f15","Type":"ContainerDied","Data":"0df9b1c6c4eb0c7a5bd5e8462acf82747d9e49d2d5df75d055dcb7120edd1c0b"} Nov 03 20:26:23.372665 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:23.372585 5111 scope.go:122] "RemoveContainer" containerID="199848a9c28a5e7c178afcb1634023e85dd9335367d3adafe46cc58b27ed605c" Nov 03 20:26:23.374367 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:23.374341 5111 generic.go:356] "Generic (PLEG): container finished" podID="78cdee3f-8ad2-4767-bf61-9c8eae1b76cc" containerID="ced7f9352f3ef9133742a6feb6aad35b7efda32886c928ea34415177cae1b8f2" exitCode=0 Nov 03 20:26:23.374467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:23.374441 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk" event={"ID":"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc","Type":"ContainerDied","Data":"ced7f9352f3ef9133742a6feb6aad35b7efda32886c928ea34415177cae1b8f2"} Nov 03 20:26:23.603615 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:23.602276 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp"] Nov 03 20:26:23.610656 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:23.610392 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a476f8-79b98cfd5f-4mfzp"] Nov 03 20:26:24.186488 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.186456 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7" path="/var/lib/kubelet/pods/bbbfe191-1dfa-4dd3-9d6c-8bbe7637a6b7/volumes" Nov 03 20:26:24.187060 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.187028 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24d736c-014e-45e0-9459-bd36e2556f15" path="/var/lib/kubelet/pods/c24d736c-014e-45e0-9459-bd36e2556f15/volumes" Nov 03 20:26:24.203484 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.203442 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2"] Nov 03 20:26:24.204994 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.204959 5111 cpu_manager.go:414] "containerMap: removing container" podUID="c24d736c-014e-45e0-9459-bd36e2556f15" containerName="payload-pre-processing" Nov 03 20:26:24.204994 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.204990 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24d736c-014e-45e0-9459-bd36e2556f15" containerName="payload-pre-processing" Nov 03 20:26:24.205365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.205344 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24d736c-014e-45e0-9459-bd36e2556f15" containerName="payload-pre-processing" Nov 03 20:26:24.225434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.224878 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj"] Nov 03 20:26:24.227045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.226997 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2" Nov 03 20:26:24.236685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.236656 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2"] Nov 03 20:26:24.236855 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.236772 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" Nov 03 20:26:24.237747 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.237660 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj"] Nov 03 20:26:24.308390 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.308354 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26fcf44-2799-4302-9dc8-84f8a3b389eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3a20436-kkptj\" (UID: \"c26fcf44-2799-4302-9dc8-84f8a3b389eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" Nov 03 20:26:24.308575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.308401 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d63738-ed89-4361-a251-f4fa9bd46253-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5668248f-862c2\" (UID: \"d2d63738-ed89-4361-a251-f4fa9bd46253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2" Nov 03 20:26:24.308575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.308419 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2kxz\" (UniqueName: \"kubernetes.io/projected/d2d63738-ed89-4361-a251-f4fa9bd46253-kube-api-access-q2kxz\") pod \"maas-api-revoke-sub-5668248f-862c2\" (UID: \"d2d63738-ed89-4361-a251-f4fa9bd46253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2" Nov 03 20:26:24.308668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.308579 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26fcf44-2799-4302-9dc8-84f8a3b389eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3a20436-kkptj\" (UID: \"c26fcf44-2799-4302-9dc8-84f8a3b389eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" Nov 03 20:26:24.308668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.308603 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d63738-ed89-4361-a251-f4fa9bd46253-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5668248f-862c2\" (UID: \"d2d63738-ed89-4361-a251-f4fa9bd46253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2" Nov 03 20:26:24.308668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.308645 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vnxq\" (UniqueName: \"kubernetes.io/projected/c26fcf44-2799-4302-9dc8-84f8a3b389eb-kube-api-access-5vnxq\") pod \"maas-api-revoke-sub-b3a20436-kkptj\" (UID: \"c26fcf44-2799-4302-9dc8-84f8a3b389eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" Nov 03 20:26:24.409990 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.409946 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26fcf44-2799-4302-9dc8-84f8a3b389eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3a20436-kkptj\" (UID: \"c26fcf44-2799-4302-9dc8-84f8a3b389eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" Nov 03 20:26:24.410180 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.410022 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d63738-ed89-4361-a251-f4fa9bd46253-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5668248f-862c2\" (UID: \"d2d63738-ed89-4361-a251-f4fa9bd46253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2" Nov 03 20:26:24.410180 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.410058 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2kxz\" (UniqueName: \"kubernetes.io/projected/d2d63738-ed89-4361-a251-f4fa9bd46253-kube-api-access-q2kxz\") pod \"maas-api-revoke-sub-5668248f-862c2\" (UID: \"d2d63738-ed89-4361-a251-f4fa9bd46253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2" Nov 03 20:26:24.410329 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.410204 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26fcf44-2799-4302-9dc8-84f8a3b389eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3a20436-kkptj\" (UID: \"c26fcf44-2799-4302-9dc8-84f8a3b389eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" Nov 03 20:26:24.410329 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.410247 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d63738-ed89-4361-a251-f4fa9bd46253-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5668248f-862c2\" (UID: \"d2d63738-ed89-4361-a251-f4fa9bd46253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2" Nov 03 20:26:24.410440 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.410349 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vnxq\" (UniqueName: \"kubernetes.io/projected/c26fcf44-2799-4302-9dc8-84f8a3b389eb-kube-api-access-5vnxq\") pod \"maas-api-revoke-sub-b3a20436-kkptj\" (UID: \"c26fcf44-2799-4302-9dc8-84f8a3b389eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" Nov 03 20:26:24.411009 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.410582 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26fcf44-2799-4302-9dc8-84f8a3b389eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3a20436-kkptj\" (UID: \"c26fcf44-2799-4302-9dc8-84f8a3b389eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" Nov 03 20:26:24.411157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.411118 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d63738-ed89-4361-a251-f4fa9bd46253-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5668248f-862c2\" (UID: \"d2d63738-ed89-4361-a251-f4fa9bd46253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2" Nov 03 20:26:24.413133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.413110 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d63738-ed89-4361-a251-f4fa9bd46253-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5668248f-862c2\" (UID: \"d2d63738-ed89-4361-a251-f4fa9bd46253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2" Nov 03 20:26:24.413499 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.413467 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26fcf44-2799-4302-9dc8-84f8a3b389eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3a20436-kkptj\" (UID: \"c26fcf44-2799-4302-9dc8-84f8a3b389eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" Nov 03 20:26:24.419143 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.419118 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vnxq\" (UniqueName: \"kubernetes.io/projected/c26fcf44-2799-4302-9dc8-84f8a3b389eb-kube-api-access-5vnxq\") pod \"maas-api-revoke-sub-b3a20436-kkptj\" (UID: \"c26fcf44-2799-4302-9dc8-84f8a3b389eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" Nov 03 20:26:24.419821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.419799 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2kxz\" (UniqueName: \"kubernetes.io/projected/d2d63738-ed89-4361-a251-f4fa9bd46253-kube-api-access-q2kxz\") pod \"maas-api-revoke-sub-5668248f-862c2\" (UID: \"d2d63738-ed89-4361-a251-f4fa9bd46253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2" Nov 03 20:26:24.564845 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.564811 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" Nov 03 20:26:24.566288 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.566032 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2" Nov 03 20:26:24.577334 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.577316 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" Nov 03 20:26:24.623374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.614384 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/384309b2-620c-49b2-818f-793652ba6cc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/384309b2-620c-49b2-818f-793652ba6cc6-maas-api-openshift-ca\") pod \"384309b2-620c-49b2-818f-793652ba6cc6\" (UID: \"384309b2-620c-49b2-818f-793652ba6cc6\") " Nov 03 20:26:24.623374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.614459 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384309b2-620c-49b2-818f-793652ba6cc6-kube-api-access-mlr5b\" (UniqueName: \"kubernetes.io/projected/384309b2-620c-49b2-818f-793652ba6cc6-kube-api-access-mlr5b\") pod \"384309b2-620c-49b2-818f-793652ba6cc6\" (UID: \"384309b2-620c-49b2-818f-793652ba6cc6\") " Nov 03 20:26:24.623374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.614492 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/384309b2-620c-49b2-818f-793652ba6cc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/384309b2-620c-49b2-818f-793652ba6cc6-maas-api-xks-ca\") pod \"384309b2-620c-49b2-818f-793652ba6cc6\" (UID: \"384309b2-620c-49b2-818f-793652ba6cc6\") " Nov 03 20:26:24.623374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.618638 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384309b2-620c-49b2-818f-793652ba6cc6-maas-api-openshift-ca" pod "384309b2-620c-49b2-818f-793652ba6cc6" (UID: "384309b2-620c-49b2-818f-793652ba6cc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:24.624868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.624831 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384309b2-620c-49b2-818f-793652ba6cc6-kube-api-access-mlr5b" pod "384309b2-620c-49b2-818f-793652ba6cc6" (UID: "384309b2-620c-49b2-818f-793652ba6cc6"). InnerVolumeSpecName "kube-api-access-mlr5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:24.633108 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.632739 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk" Nov 03 20:26:24.637510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.637482 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/384309b2-620c-49b2-818f-793652ba6cc6-maas-api-xks-ca" pod "384309b2-620c-49b2-818f-793652ba6cc6" (UID: "384309b2-620c-49b2-818f-793652ba6cc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:24.749634 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.744449 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlr5b\" (UniqueName: \"kubernetes.io/projected/384309b2-620c-49b2-818f-793652ba6cc6-kube-api-access-mlr5b\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.749634 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.744488 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/384309b2-620c-49b2-818f-793652ba6cc6-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.749634 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.744504 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/384309b2-620c-49b2-818f-793652ba6cc6-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.845128 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.845091 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-maas-api-xks-ca\") pod \"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc\" (UID: \"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc\") " Nov 03 20:26:24.845328 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.845236 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-maas-api-openshift-ca\") pod \"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc\" (UID: \"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc\") " Nov 03 20:26:24.845420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.845408 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-kube-api-access-jwdhl\" (UniqueName: \"kubernetes.io/projected/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-kube-api-access-jwdhl\") pod \"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc\" (UID: \"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc\") " Nov 03 20:26:24.846124 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.845959 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-maas-api-openshift-ca" pod "78cdee3f-8ad2-4767-bf61-9c8eae1b76cc" (UID: "78cdee3f-8ad2-4767-bf61-9c8eae1b76cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:24.847995 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.847954 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-maas-api-xks-ca" pod "78cdee3f-8ad2-4767-bf61-9c8eae1b76cc" (UID: "78cdee3f-8ad2-4767-bf61-9c8eae1b76cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:24.848089 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.848012 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-kube-api-access-jwdhl" pod "78cdee3f-8ad2-4767-bf61-9c8eae1b76cc" (UID: "78cdee3f-8ad2-4767-bf61-9c8eae1b76cc"). InnerVolumeSpecName "kube-api-access-jwdhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:24.879675 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:26:24.879639 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26fcf44_2799_4302_9dc8_84f8a3b389eb.slice/crio-4fe40582e47104dd5dfee3170b90827c84782c8e22f074aa941ca808b6e2239a WatchSource:0}: Error finding container 4fe40582e47104dd5dfee3170b90827c84782c8e22f074aa941ca808b6e2239a: Status 404 returned error can't find the container with id 4fe40582e47104dd5dfee3170b90827c84782c8e22f074aa941ca808b6e2239a Nov 03 20:26:24.880605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.880576 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj"] Nov 03 20:26:24.927416 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.927379 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2"] Nov 03 20:26:24.948051 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.948015 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwdhl\" (UniqueName: \"kubernetes.io/projected/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-kube-api-access-jwdhl\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.948209 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.948056 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.948209 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:24.948070 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:25.395133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:25.395104 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk" Nov 03 20:26:25.395293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:25.395121 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk" event={"ID":"78cdee3f-8ad2-4767-bf61-9c8eae1b76cc","Type":"ContainerDied","Data":"2357f9e4de118ef675b6a242a0b94a91e746eff13e053cf76dbd641419d69986"} Nov 03 20:26:25.395293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:25.395206 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2357f9e4de118ef675b6a242a0b94a91e746eff13e053cf76dbd641419d69986" Nov 03 20:26:25.396576 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:25.396552 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2" event={"ID":"d2d63738-ed89-4361-a251-f4fa9bd46253","Type":"ContainerStarted","Data":"06772a3741ade7a7a89773ab09e08cf04af658b5e960d1212964ec757a29312b"} Nov 03 20:26:25.398092 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:25.398068 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" Nov 03 20:26:25.398092 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:25.398077 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx" event={"ID":"384309b2-620c-49b2-818f-793652ba6cc6","Type":"ContainerDied","Data":"e2f570950314d68ad6478fe545fd5e5b8b74c330f962b4350846146a43272abb"} Nov 03 20:26:25.398253 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:25.398112 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2f570950314d68ad6478fe545fd5e5b8b74c330f962b4350846146a43272abb" Nov 03 20:26:25.399813 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:25.399785 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" event={"ID":"c26fcf44-2799-4302-9dc8-84f8a3b389eb","Type":"ContainerStarted","Data":"f2f0c8579dab16dcd9f0332d0947401fa84bd99fbece61b45ad9e72b39fc5728"} Nov 03 20:26:25.399933 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:25.399841 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" event={"ID":"c26fcf44-2799-4302-9dc8-84f8a3b389eb","Type":"ContainerStarted","Data":"4fe40582e47104dd5dfee3170b90827c84782c8e22f074aa941ca808b6e2239a"} Nov 03 20:26:25.423585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:25.420229 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" podStartSLOduration=1.4202177489999999 podStartE2EDuration="1.420217749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:25.416357105 +0000 UTC m=+2815.378735544" watchObservedRunningTime="2026-11-03 20:26:25.420217749 +0000 UTC m=+2815.382596186" Nov 03 20:26:26.409774 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:26.409740 5111 generic.go:356] "Generic (PLEG): container finished" podID="c26fcf44-2799-4302-9dc8-84f8a3b389eb" containerID="f2f0c8579dab16dcd9f0332d0947401fa84bd99fbece61b45ad9e72b39fc5728" exitCode=0 Nov 03 20:26:26.410260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:26.409824 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" event={"ID":"c26fcf44-2799-4302-9dc8-84f8a3b389eb","Type":"ContainerDied","Data":"f2f0c8579dab16dcd9f0332d0947401fa84bd99fbece61b45ad9e72b39fc5728"} Nov 03 20:26:26.412018 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:26.411992 5111 generic.go:356] "Generic (PLEG): container finished" podID="d2d63738-ed89-4361-a251-f4fa9bd46253" containerID="b79952d3db616b5c53d16e6c9920607d29c9afa644fdd2e3a657eb74ca2bacde" exitCode=0 Nov 03 20:26:26.412136 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:26.412076 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2" event={"ID":"d2d63738-ed89-4361-a251-f4fa9bd46253","Type":"ContainerDied","Data":"b79952d3db616b5c53d16e6c9920607d29c9afa644fdd2e3a657eb74ca2bacde"} Nov 03 20:26:26.751058 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:26.750961 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" podUID="55381c03-06d5-4cdb-841f-265b16b1d8f8" containerName="payload-pre-processing" containerID="cri-o://ad865cc5d91a551c27a0b208cb906f2c38cb9852ca042ddf2739fc65a9c78e42" gracePeriod=25 Nov 03 20:26:27.062445 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.062417 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" Nov 03 20:26:27.102251 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.102216 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55381c03-06d5-4cdb-841f-265b16b1d8f8-kube-api-access-sw9hd\" (UniqueName: \"kubernetes.io/projected/55381c03-06d5-4cdb-841f-265b16b1d8f8-kube-api-access-sw9hd\") pod \"55381c03-06d5-4cdb-841f-265b16b1d8f8\" (UID: \"55381c03-06d5-4cdb-841f-265b16b1d8f8\") " Nov 03 20:26:27.102443 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.102273 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55381c03-06d5-4cdb-841f-265b16b1d8f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55381c03-06d5-4cdb-841f-265b16b1d8f8-plugins-config-volume\") pod \"55381c03-06d5-4cdb-841f-265b16b1d8f8\" (UID: \"55381c03-06d5-4cdb-841f-265b16b1d8f8\") " Nov 03 20:26:27.103261 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.103227 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55381c03-06d5-4cdb-841f-265b16b1d8f8-plugins-config-volume" pod "55381c03-06d5-4cdb-841f-265b16b1d8f8" (UID: "55381c03-06d5-4cdb-841f-265b16b1d8f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:27.104962 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.104932 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55381c03-06d5-4cdb-841f-265b16b1d8f8-kube-api-access-sw9hd" pod "55381c03-06d5-4cdb-841f-265b16b1d8f8" (UID: "55381c03-06d5-4cdb-841f-265b16b1d8f8"). InnerVolumeSpecName "kube-api-access-sw9hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:27.204211 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.204174 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw9hd\" (UniqueName: \"kubernetes.io/projected/55381c03-06d5-4cdb-841f-265b16b1d8f8-kube-api-access-sw9hd\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:27.204211 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.204210 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55381c03-06d5-4cdb-841f-265b16b1d8f8-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:27.422169 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.422132 5111 generic.go:356] "Generic (PLEG): container finished" podID="55381c03-06d5-4cdb-841f-265b16b1d8f8" containerID="ad865cc5d91a551c27a0b208cb906f2c38cb9852ca042ddf2739fc65a9c78e42" exitCode=0 Nov 03 20:26:27.422744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.422183 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" Nov 03 20:26:27.422744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.422218 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" event={"ID":"55381c03-06d5-4cdb-841f-265b16b1d8f8","Type":"ContainerDied","Data":"ad865cc5d91a551c27a0b208cb906f2c38cb9852ca042ddf2739fc65a9c78e42"} Nov 03 20:26:27.422744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.422287 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5" event={"ID":"55381c03-06d5-4cdb-841f-265b16b1d8f8","Type":"ContainerDied","Data":"0dfbbdac1b08627a09ec804b18f557c0bfe6e0469d8b1bda02cc945ae13ce582"} Nov 03 20:26:27.422744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.422322 5111 scope.go:122] "RemoveContainer" containerID="ad865cc5d91a551c27a0b208cb906f2c38cb9852ca042ddf2739fc65a9c78e42" Nov 03 20:26:27.437901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.437878 5111 scope.go:122] "RemoveContainer" containerID="ad865cc5d91a551c27a0b208cb906f2c38cb9852ca042ddf2739fc65a9c78e42" Nov 03 20:26:27.438322 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:27.438252 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad865cc5d91a551c27a0b208cb906f2c38cb9852ca042ddf2739fc65a9c78e42\": container with ID starting with ad865cc5d91a551c27a0b208cb906f2c38cb9852ca042ddf2739fc65a9c78e42 not found: ID does not exist" containerID="ad865cc5d91a551c27a0b208cb906f2c38cb9852ca042ddf2739fc65a9c78e42" Nov 03 20:26:27.438509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.438474 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad865cc5d91a551c27a0b208cb906f2c38cb9852ca042ddf2739fc65a9c78e42"} err="failed to get container status \"ad865cc5d91a551c27a0b208cb906f2c38cb9852ca042ddf2739fc65a9c78e42\": rpc error: code = NotFound desc = could not find container \"ad865cc5d91a551c27a0b208cb906f2c38cb9852ca042ddf2739fc65a9c78e42\": container with ID starting with ad865cc5d91a551c27a0b208cb906f2c38cb9852ca042ddf2739fc65a9c78e42 not found: ID does not exist" Nov 03 20:26:27.454358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.454321 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5"] Nov 03 20:26:27.459167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.459135 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-0ed729-68454bb49f-swsd5"] Nov 03 20:26:27.608547 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.608510 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2" Nov 03 20:26:27.650312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.650281 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" Nov 03 20:26:27.711928 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.711847 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c26fcf44-2799-4302-9dc8-84f8a3b389eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26fcf44-2799-4302-9dc8-84f8a3b389eb-maas-api-openshift-ca\") pod \"c26fcf44-2799-4302-9dc8-84f8a3b389eb\" (UID: \"c26fcf44-2799-4302-9dc8-84f8a3b389eb\") " Nov 03 20:26:27.711928 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.711909 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2d63738-ed89-4361-a251-f4fa9bd46253-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d63738-ed89-4361-a251-f4fa9bd46253-maas-api-xks-ca\") pod \"d2d63738-ed89-4361-a251-f4fa9bd46253\" (UID: \"d2d63738-ed89-4361-a251-f4fa9bd46253\") " Nov 03 20:26:27.712174 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.712010 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d63738-ed89-4361-a251-f4fa9bd46253-kube-api-access-q2kxz\" (UniqueName: \"kubernetes.io/projected/d2d63738-ed89-4361-a251-f4fa9bd46253-kube-api-access-q2kxz\") pod \"d2d63738-ed89-4361-a251-f4fa9bd46253\" (UID: \"d2d63738-ed89-4361-a251-f4fa9bd46253\") " Nov 03 20:26:27.712174 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.712060 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c26fcf44-2799-4302-9dc8-84f8a3b389eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26fcf44-2799-4302-9dc8-84f8a3b389eb-maas-api-xks-ca\") pod \"c26fcf44-2799-4302-9dc8-84f8a3b389eb\" (UID: \"c26fcf44-2799-4302-9dc8-84f8a3b389eb\") " Nov 03 20:26:27.712298 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.712244 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2d63738-ed89-4361-a251-f4fa9bd46253-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d63738-ed89-4361-a251-f4fa9bd46253-maas-api-openshift-ca\") pod \"d2d63738-ed89-4361-a251-f4fa9bd46253\" (UID: \"d2d63738-ed89-4361-a251-f4fa9bd46253\") " Nov 03 20:26:27.712298 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.712265 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26fcf44-2799-4302-9dc8-84f8a3b389eb-maas-api-openshift-ca" pod "c26fcf44-2799-4302-9dc8-84f8a3b389eb" (UID: "c26fcf44-2799-4302-9dc8-84f8a3b389eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:27.712298 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.712282 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26fcf44-2799-4302-9dc8-84f8a3b389eb-kube-api-access-5vnxq\" (UniqueName: \"kubernetes.io/projected/c26fcf44-2799-4302-9dc8-84f8a3b389eb-kube-api-access-5vnxq\") pod \"c26fcf44-2799-4302-9dc8-84f8a3b389eb\" (UID: \"c26fcf44-2799-4302-9dc8-84f8a3b389eb\") " Nov 03 20:26:27.712783 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.712722 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d63738-ed89-4361-a251-f4fa9bd46253-maas-api-openshift-ca" pod "d2d63738-ed89-4361-a251-f4fa9bd46253" (UID: "d2d63738-ed89-4361-a251-f4fa9bd46253"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:27.713092 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.713066 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26fcf44-2799-4302-9dc8-84f8a3b389eb-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:27.713173 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.713097 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d63738-ed89-4361-a251-f4fa9bd46253-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:27.714684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.714565 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26fcf44-2799-4302-9dc8-84f8a3b389eb-kube-api-access-5vnxq" pod "c26fcf44-2799-4302-9dc8-84f8a3b389eb" (UID: "c26fcf44-2799-4302-9dc8-84f8a3b389eb"). InnerVolumeSpecName "kube-api-access-5vnxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:27.714790 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.714693 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d63738-ed89-4361-a251-f4fa9bd46253-maas-api-xks-ca" pod "d2d63738-ed89-4361-a251-f4fa9bd46253" (UID: "d2d63738-ed89-4361-a251-f4fa9bd46253"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:27.714960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.714938 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c26fcf44-2799-4302-9dc8-84f8a3b389eb-maas-api-xks-ca" pod "c26fcf44-2799-4302-9dc8-84f8a3b389eb" (UID: "c26fcf44-2799-4302-9dc8-84f8a3b389eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:27.716225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.716204 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d63738-ed89-4361-a251-f4fa9bd46253-kube-api-access-q2kxz" pod "d2d63738-ed89-4361-a251-f4fa9bd46253" (UID: "d2d63738-ed89-4361-a251-f4fa9bd46253"). InnerVolumeSpecName "kube-api-access-q2kxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:27.826691 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.825775 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d63738-ed89-4361-a251-f4fa9bd46253-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:27.827025 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.826940 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2kxz\" (UniqueName: \"kubernetes.io/projected/d2d63738-ed89-4361-a251-f4fa9bd46253-kube-api-access-q2kxz\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:27.827025 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.826972 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26fcf44-2799-4302-9dc8-84f8a3b389eb-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:27.827025 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:27.826991 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vnxq\" (UniqueName: \"kubernetes.io/projected/c26fcf44-2799-4302-9dc8-84f8a3b389eb-kube-api-access-5vnxq\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:28.186811 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:28.186776 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55381c03-06d5-4cdb-841f-265b16b1d8f8" path="/var/lib/kubelet/pods/55381c03-06d5-4cdb-841f-265b16b1d8f8/volumes" Nov 03 20:26:28.433169 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:28.433139 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" Nov 03 20:26:28.433642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:28.433169 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj" event={"ID":"c26fcf44-2799-4302-9dc8-84f8a3b389eb","Type":"ContainerDied","Data":"4fe40582e47104dd5dfee3170b90827c84782c8e22f074aa941ca808b6e2239a"} Nov 03 20:26:28.433642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:28.433220 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fe40582e47104dd5dfee3170b90827c84782c8e22f074aa941ca808b6e2239a" Nov 03 20:26:28.435076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:28.435043 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2" event={"ID":"d2d63738-ed89-4361-a251-f4fa9bd46253","Type":"ContainerDied","Data":"06772a3741ade7a7a89773ab09e08cf04af658b5e960d1212964ec757a29312b"} Nov 03 20:26:28.435232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:28.435079 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2" Nov 03 20:26:28.435232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:28.435099 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06772a3741ade7a7a89773ab09e08cf04af658b5e960d1212964ec757a29312b" Nov 03 20:26:30.370920 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:30.370882 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl"] Nov 03 20:26:30.371734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:30.371676 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" podUID="f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" containerName="istio-proxy" containerID="cri-o://8cbe909a2f6df9ace58ef88b130ed5d0c0d84709df84c735a7ec87bf0a7bf177" gracePeriod=30 Nov 03 20:26:30.380940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:30.380906 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx"] Nov 03 20:26:30.381425 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:30.381363 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" containerID="cri-o://e1d8e1211010cf0fc71792f2cd58c8a5a95de6b2311ec1d8e5dbaa16424ad724" gracePeriod=30 Nov 03 20:26:30.771590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:30.771483 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx"] Nov 03 20:26:30.784175 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:30.784138 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs"] Nov 03 20:26:31.000953 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.000916 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq"] Nov 03 20:26:31.002692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.002665 5111 cpu_manager.go:414] "containerMap: removing container" podUID="55381c03-06d5-4cdb-841f-265b16b1d8f8" containerName="payload-pre-processing" Nov 03 20:26:31.002865 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.002847 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55381c03-06d5-4cdb-841f-265b16b1d8f8" containerName="payload-pre-processing" Nov 03 20:26:31.003309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.003289 5111 cpu_manager.go:414] "containerMap: removing container" podUID="78cdee3f-8ad2-4767-bf61-9c8eae1b76cc" containerName="revoke-keys" Nov 03 20:26:31.003473 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.003457 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78cdee3f-8ad2-4767-bf61-9c8eae1b76cc" containerName="revoke-keys" Nov 03 20:26:31.004265 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.004066 5111 cpu_manager.go:414] "containerMap: removing container" podUID="384309b2-620c-49b2-818f-793652ba6cc6" containerName="revoke-keys" Nov 03 20:26:31.004421 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.004404 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384309b2-620c-49b2-818f-793652ba6cc6" containerName="revoke-keys" Nov 03 20:26:31.004550 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.004522 5111 cpu_manager.go:414] "containerMap: removing container" podUID="d2d63738-ed89-4361-a251-f4fa9bd46253" containerName="revoke-keys" Nov 03 20:26:31.005101 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.005075 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d63738-ed89-4361-a251-f4fa9bd46253" containerName="revoke-keys" Nov 03 20:26:31.005248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.005233 5111 cpu_manager.go:414] "containerMap: removing container" podUID="c26fcf44-2799-4302-9dc8-84f8a3b389eb" containerName="revoke-keys" Nov 03 20:26:31.005352 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.005339 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c26fcf44-2799-4302-9dc8-84f8a3b389eb" containerName="revoke-keys" Nov 03 20:26:31.005673 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.005655 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="78cdee3f-8ad2-4767-bf61-9c8eae1b76cc" containerName="revoke-keys" Nov 03 20:26:31.005877 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.005836 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="384309b2-620c-49b2-818f-793652ba6cc6" containerName="revoke-keys" Nov 03 20:26:31.005980 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.005968 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d63738-ed89-4361-a251-f4fa9bd46253" containerName="revoke-keys" Nov 03 20:26:31.006060 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.006050 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="c26fcf44-2799-4302-9dc8-84f8a3b389eb" containerName="revoke-keys" Nov 03 20:26:31.006133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.006124 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="55381c03-06d5-4cdb-841f-265b16b1d8f8" containerName="payload-pre-processing" Nov 03 20:26:31.026608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.019132 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" Nov 03 20:26:31.074556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.060919 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:26:31.074556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.061656 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:26:31.074556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.061806 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:26:31.074556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.061897 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:26:31.074556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.064148 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq"] Nov 03 20:26:31.137632 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.137591 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnnws\" (UniqueName: \"kubernetes.io/projected/38304073-d13f-4d9c-ac90-81ba88cb910b-kube-api-access-lnnws\") pod \"maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq\" (UID: \"38304073-d13f-4d9c-ac90-81ba88cb910b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" Nov 03 20:26:31.137840 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.137754 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38304073-d13f-4d9c-ac90-81ba88cb910b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq\" (UID: \"38304073-d13f-4d9c-ac90-81ba88cb910b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" Nov 03 20:26:31.137906 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.137847 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38304073-d13f-4d9c-ac90-81ba88cb910b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq\" (UID: \"38304073-d13f-4d9c-ac90-81ba88cb910b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" Nov 03 20:26:31.239215 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.239177 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38304073-d13f-4d9c-ac90-81ba88cb910b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq\" (UID: \"38304073-d13f-4d9c-ac90-81ba88cb910b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" Nov 03 20:26:31.239464 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.239304 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38304073-d13f-4d9c-ac90-81ba88cb910b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq\" (UID: \"38304073-d13f-4d9c-ac90-81ba88cb910b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" Nov 03 20:26:31.239583 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.239473 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnnws\" (UniqueName: \"kubernetes.io/projected/38304073-d13f-4d9c-ac90-81ba88cb910b-kube-api-access-lnnws\") pod \"maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq\" (UID: \"38304073-d13f-4d9c-ac90-81ba88cb910b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" Nov 03 20:26:31.240117 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.240079 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38304073-d13f-4d9c-ac90-81ba88cb910b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq\" (UID: \"38304073-d13f-4d9c-ac90-81ba88cb910b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" Nov 03 20:26:31.242243 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.242218 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38304073-d13f-4d9c-ac90-81ba88cb910b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq\" (UID: \"38304073-d13f-4d9c-ac90-81ba88cb910b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" Nov 03 20:26:31.247751 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.247730 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnnws\" (UniqueName: \"kubernetes.io/projected/38304073-d13f-4d9c-ac90-81ba88cb910b-kube-api-access-lnnws\") pod \"maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq\" (UID: \"38304073-d13f-4d9c-ac90-81ba88cb910b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" Nov 03 20:26:31.320909 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.320809 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" containerID="cri-o://a76d15f0aa5639d6ad292234693d08d5e96a18dce9d700d46126dd62f10fad9b" gracePeriod=45 Nov 03 20:26:31.439411 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.439378 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" Nov 03 20:26:31.480183 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.480149 5111 generic.go:356] "Generic (PLEG): container finished" podID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerID="a76d15f0aa5639d6ad292234693d08d5e96a18dce9d700d46126dd62f10fad9b" exitCode=0 Nov 03 20:26:31.480631 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.480602 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" event={"ID":"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8","Type":"ContainerDied","Data":"a76d15f0aa5639d6ad292234693d08d5e96a18dce9d700d46126dd62f10fad9b"} Nov 03 20:26:31.575159 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.572374 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq"] Nov 03 20:26:31.690946 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.690912 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:26:31.770992 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.770955 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-model-cache\") pod \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " Nov 03 20:26:31.771207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.771040 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-tls-certs\") pod \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " Nov 03 20:26:31.771207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.771127 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-kube-api-access-5gjh9\" (UniqueName: \"kubernetes.io/projected/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-kube-api-access-5gjh9\") pod \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " Nov 03 20:26:31.772309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.771601 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-home\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-home\") pod \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " Nov 03 20:26:31.772309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.771672 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-tmp-dir\") pod \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " Nov 03 20:26:31.772309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.771805 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-dshm\") pod \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\" (UID: \"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8\") " Nov 03 20:26:31.772309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.771881 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-model-cache" pod "3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" (UID: "3bd3b520-ce5d-4bfd-9fcf-89b385318ad8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:31.772548 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.772513 5111 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-model-cache\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:31.773575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.772862 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-home" pod "3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" (UID: "3bd3b520-ce5d-4bfd-9fcf-89b385318ad8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:31.773575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.773032 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-tmp-dir" pod "3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" (UID: "3bd3b520-ce5d-4bfd-9fcf-89b385318ad8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:31.774142 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.774115 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-kube-api-access-5gjh9" pod "3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" (UID: "3bd3b520-ce5d-4bfd-9fcf-89b385318ad8"). InnerVolumeSpecName "kube-api-access-5gjh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:31.774253 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.774223 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-tls-certs" pod "3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" (UID: "3bd3b520-ce5d-4bfd-9fcf-89b385318ad8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:31.775383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.775361 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-dshm" pod "3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" (UID: "3bd3b520-ce5d-4bfd-9fcf-89b385318ad8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:31.806968 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.806942 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq"] Nov 03 20:26:31.808443 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:26:31.808418 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38304073_d13f_4d9c_ac90_81ba88cb910b.slice/crio-596c6bdf8fd1dd466fc5dc56cebb3c0da70d2099dc76a5eea0ecd18135ddb93a WatchSource:0}: Error finding container 596c6bdf8fd1dd466fc5dc56cebb3c0da70d2099dc76a5eea0ecd18135ddb93a: Status 404 returned error can't find the container with id 596c6bdf8fd1dd466fc5dc56cebb3c0da70d2099dc76a5eea0ecd18135ddb93a Nov 03 20:26:31.874120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.874083 5111 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-dshm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:31.874120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.874122 5111 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-tls-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:31.874347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.874137 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gjh9\" (UniqueName: \"kubernetes.io/projected/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-kube-api-access-5gjh9\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:31.874347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.874154 5111 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-home\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:31.874347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:31.874167 5111 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8-tmp-dir\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:32.141014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.140969 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" podUID="af7b24bc-adce-4c7a-b2be-5844a4d44675" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:26:32.495352 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.495300 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" event={"ID":"3bd3b520-ce5d-4bfd-9fcf-89b385318ad8","Type":"ContainerDied","Data":"a01b53550954fdf4c9031401077ea07cd0e1459c7514c9d3f163f8c426bdb68f"} Nov 03 20:26:32.495879 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.495388 5111 scope.go:122] "RemoveContainer" containerID="a76d15f0aa5639d6ad292234693d08d5e96a18dce9d700d46126dd62f10fad9b" Nov 03 20:26:32.495879 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.495657 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg" Nov 03 20:26:32.500757 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.500731 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" event={"ID":"38304073-d13f-4d9c-ac90-81ba88cb910b","Type":"ContainerStarted","Data":"2a4ef095be3a6770ff116ea63c76e696ba321c8e5fef21814e76103d62d2f7aa"} Nov 03 20:26:32.501000 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.500980 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" event={"ID":"38304073-d13f-4d9c-ac90-81ba88cb910b","Type":"ContainerStarted","Data":"596c6bdf8fd1dd466fc5dc56cebb3c0da70d2099dc76a5eea0ecd18135ddb93a"} Nov 03 20:26:32.503439 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.503418 5111 generic.go:356] "Generic (PLEG): container finished" podID="af7b24bc-adce-4c7a-b2be-5844a4d44675" containerID="0002aa5a90ea141e32e45508480c90a1fb37a048cebf58148e9438179853dbb3" exitCode=0 Nov 03 20:26:32.503738 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.503714 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" event={"ID":"af7b24bc-adce-4c7a-b2be-5844a4d44675","Type":"ContainerDied","Data":"0002aa5a90ea141e32e45508480c90a1fb37a048cebf58148e9438179853dbb3"} Nov 03 20:26:32.526397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.526358 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg"] Nov 03 20:26:32.532747 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.532718 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/simulator-w6-0ed729-kserve-7b95b48f76-9d4tg"] Nov 03 20:26:32.535901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.535851 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" podStartSLOduration=2.535835484 podStartE2EDuration="2.535835484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:32.531352794 +0000 UTC m=+2822.493731235" watchObservedRunningTime="2026-11-03 20:26:32.535835484 +0000 UTC m=+2822.498213923" Nov 03 20:26:32.560948 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.560920 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" Nov 03 20:26:32.590211 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.590175 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af7b24bc-adce-4c7a-b2be-5844a4d44675-kube-api-access-876gk\" (UniqueName: \"kubernetes.io/projected/af7b24bc-adce-4c7a-b2be-5844a4d44675-kube-api-access-876gk\") pod \"af7b24bc-adce-4c7a-b2be-5844a4d44675\" (UID: \"af7b24bc-adce-4c7a-b2be-5844a4d44675\") " Nov 03 20:26:32.590387 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.590369 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af7b24bc-adce-4c7a-b2be-5844a4d44675-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af7b24bc-adce-4c7a-b2be-5844a4d44675-plugins-config-volume\") pod \"af7b24bc-adce-4c7a-b2be-5844a4d44675\" (UID: \"af7b24bc-adce-4c7a-b2be-5844a4d44675\") " Nov 03 20:26:32.591021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.590991 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7b24bc-adce-4c7a-b2be-5844a4d44675-plugins-config-volume" pod "af7b24bc-adce-4c7a-b2be-5844a4d44675" (UID: "af7b24bc-adce-4c7a-b2be-5844a4d44675"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:32.597794 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.597763 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af7b24bc-adce-4c7a-b2be-5844a4d44675-kube-api-access-876gk" pod "af7b24bc-adce-4c7a-b2be-5844a4d44675" (UID: "af7b24bc-adce-4c7a-b2be-5844a4d44675"). InnerVolumeSpecName "kube-api-access-876gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:32.692566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.692479 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-876gk\" (UniqueName: \"kubernetes.io/projected/af7b24bc-adce-4c7a-b2be-5844a4d44675-kube-api-access-876gk\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:32.692566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:32.692513 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af7b24bc-adce-4c7a-b2be-5844a4d44675-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:33.170728 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.170684 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:26:33.170945 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.170768 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" podUID="f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:26:33.514985 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.514901 5111 generic.go:356] "Generic (PLEG): container finished" podID="38304073-d13f-4d9c-ac90-81ba88cb910b" containerID="2a4ef095be3a6770ff116ea63c76e696ba321c8e5fef21814e76103d62d2f7aa" exitCode=0 Nov 03 20:26:33.515469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.514987 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" event={"ID":"38304073-d13f-4d9c-ac90-81ba88cb910b","Type":"ContainerDied","Data":"2a4ef095be3a6770ff116ea63c76e696ba321c8e5fef21814e76103d62d2f7aa"} Nov 03 20:26:33.516833 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.516811 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" Nov 03 20:26:33.516972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.516845 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8" event={"ID":"af7b24bc-adce-4c7a-b2be-5844a4d44675","Type":"ContainerDied","Data":"f162fea08b5473e83ff73dd3e2c7c61bd976ce00190617df5a87df727d805d47"} Nov 03 20:26:33.516972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.516903 5111 scope.go:122] "RemoveContainer" containerID="0002aa5a90ea141e32e45508480c90a1fb37a048cebf58148e9438179853dbb3" Nov 03 20:26:33.553122 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.553092 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8"] Nov 03 20:26:33.559528 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.559503 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22a567-66dcddf555-np4j8"] Nov 03 20:26:33.875738 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.875707 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn"] Nov 03 20:26:33.876351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.876335 5111 cpu_manager.go:414] "containerMap: removing container" podUID="af7b24bc-adce-4c7a-b2be-5844a4d44675" containerName="payload-pre-processing" Nov 03 20:26:33.876420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.876352 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af7b24bc-adce-4c7a-b2be-5844a4d44675" containerName="payload-pre-processing" Nov 03 20:26:33.876420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.876380 5111 cpu_manager.go:414] "containerMap: removing container" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" Nov 03 20:26:33.876420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.876386 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" Nov 03 20:26:33.876517 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.876467 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="af7b24bc-adce-4c7a-b2be-5844a4d44675" containerName="payload-pre-processing" Nov 03 20:26:33.876517 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.876482 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" containerName="main" Nov 03 20:26:33.881434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.881413 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn" Nov 03 20:26:33.888724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.888698 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn"] Nov 03 20:26:33.913692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.913657 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfeea28-db58-4740-b089-9d8eb0dfece7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn\" (UID: \"5bfeea28-db58-4740-b089-9d8eb0dfece7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn" Nov 03 20:26:33.913851 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.913814 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfeea28-db58-4740-b089-9d8eb0dfece7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn\" (UID: \"5bfeea28-db58-4740-b089-9d8eb0dfece7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn" Nov 03 20:26:33.913936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:33.913859 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx5kj\" (UniqueName: \"kubernetes.io/projected/5bfeea28-db58-4740-b089-9d8eb0dfece7-kube-api-access-gx5kj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn\" (UID: \"5bfeea28-db58-4740-b089-9d8eb0dfece7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn" Nov 03 20:26:34.015722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.015681 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfeea28-db58-4740-b089-9d8eb0dfece7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn\" (UID: \"5bfeea28-db58-4740-b089-9d8eb0dfece7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn" Nov 03 20:26:34.015917 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.015834 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfeea28-db58-4740-b089-9d8eb0dfece7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn\" (UID: \"5bfeea28-db58-4740-b089-9d8eb0dfece7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn" Nov 03 20:26:34.015917 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.015885 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx5kj\" (UniqueName: \"kubernetes.io/projected/5bfeea28-db58-4740-b089-9d8eb0dfece7-kube-api-access-gx5kj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn\" (UID: \"5bfeea28-db58-4740-b089-9d8eb0dfece7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn" Nov 03 20:26:34.016451 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.016422 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfeea28-db58-4740-b089-9d8eb0dfece7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn\" (UID: \"5bfeea28-db58-4740-b089-9d8eb0dfece7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn" Nov 03 20:26:34.018231 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.018211 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfeea28-db58-4740-b089-9d8eb0dfece7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn\" (UID: \"5bfeea28-db58-4740-b089-9d8eb0dfece7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn" Nov 03 20:26:34.027267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.027241 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx5kj\" (UniqueName: \"kubernetes.io/projected/5bfeea28-db58-4740-b089-9d8eb0dfece7-kube-api-access-gx5kj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn\" (UID: \"5bfeea28-db58-4740-b089-9d8eb0dfece7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn" Nov 03 20:26:34.200054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.199972 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd3b520-ce5d-4bfd-9fcf-89b385318ad8" path="/var/lib/kubelet/pods/3bd3b520-ce5d-4bfd-9fcf-89b385318ad8/volumes" Nov 03 20:26:34.200597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.200573 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af7b24bc-adce-4c7a-b2be-5844a4d44675" path="/var/lib/kubelet/pods/af7b24bc-adce-4c7a-b2be-5844a4d44675/volumes" Nov 03 20:26:34.203717 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.203689 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn" Nov 03 20:26:34.488566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.488526 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn"] Nov 03 20:26:34.489883 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:26:34.489854 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfeea28_db58_4740_b089_9d8eb0dfece7.slice/crio-4f54bb7ae84f0e6d8757e824e5e2dab2ce1c36b8084afef16e63d0f053c8e42e WatchSource:0}: Error finding container 4f54bb7ae84f0e6d8757e824e5e2dab2ce1c36b8084afef16e63d0f053c8e42e: Status 404 returned error can't find the container with id 4f54bb7ae84f0e6d8757e824e5e2dab2ce1c36b8084afef16e63d0f053c8e42e Nov 03 20:26:34.532486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.532426 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn" event={"ID":"5bfeea28-db58-4740-b089-9d8eb0dfece7","Type":"ContainerStarted","Data":"4f54bb7ae84f0e6d8757e824e5e2dab2ce1c36b8084afef16e63d0f053c8e42e"} Nov 03 20:26:34.810246 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.810214 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" Nov 03 20:26:34.929347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.929306 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38304073-d13f-4d9c-ac90-81ba88cb910b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38304073-d13f-4d9c-ac90-81ba88cb910b-maas-api-xks-ca\") pod \"38304073-d13f-4d9c-ac90-81ba88cb910b\" (UID: \"38304073-d13f-4d9c-ac90-81ba88cb910b\") " Nov 03 20:26:34.929510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.929477 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38304073-d13f-4d9c-ac90-81ba88cb910b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38304073-d13f-4d9c-ac90-81ba88cb910b-maas-api-openshift-ca\") pod \"38304073-d13f-4d9c-ac90-81ba88cb910b\" (UID: \"38304073-d13f-4d9c-ac90-81ba88cb910b\") " Nov 03 20:26:34.929623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.929597 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38304073-d13f-4d9c-ac90-81ba88cb910b-kube-api-access-lnnws\" (UniqueName: \"kubernetes.io/projected/38304073-d13f-4d9c-ac90-81ba88cb910b-kube-api-access-lnnws\") pod \"38304073-d13f-4d9c-ac90-81ba88cb910b\" (UID: \"38304073-d13f-4d9c-ac90-81ba88cb910b\") " Nov 03 20:26:34.929848 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.929815 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38304073-d13f-4d9c-ac90-81ba88cb910b-maas-api-openshift-ca" pod "38304073-d13f-4d9c-ac90-81ba88cb910b" (UID: "38304073-d13f-4d9c-ac90-81ba88cb910b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:34.930121 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.930099 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38304073-d13f-4d9c-ac90-81ba88cb910b-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:34.945292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.945257 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38304073-d13f-4d9c-ac90-81ba88cb910b-maas-api-xks-ca" pod "38304073-d13f-4d9c-ac90-81ba88cb910b" (UID: "38304073-d13f-4d9c-ac90-81ba88cb910b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:34.945425 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:34.945320 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38304073-d13f-4d9c-ac90-81ba88cb910b-kube-api-access-lnnws" pod "38304073-d13f-4d9c-ac90-81ba88cb910b" (UID: "38304073-d13f-4d9c-ac90-81ba88cb910b"). InnerVolumeSpecName "kube-api-access-lnnws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:35.037614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.032731 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38304073-d13f-4d9c-ac90-81ba88cb910b-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.037614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.032771 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnnws\" (UniqueName: \"kubernetes.io/projected/38304073-d13f-4d9c-ac90-81ba88cb910b-kube-api-access-lnnws\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.116054 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:35.116009 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:26:35.194204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.194169 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx"] Nov 03 20:26:35.198774 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.198745 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27d3ad-mvtpx"] Nov 03 20:26:35.549025 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.548772 5111 generic.go:356] "Generic (PLEG): container finished" podID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerID="e1d8e1211010cf0fc71792f2cd58c8a5a95de6b2311ec1d8e5dbaa16424ad724" exitCode=0 Nov 03 20:26:35.549025 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.548924 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" event={"ID":"027fe01b-c13b-4f05-babf-c440793dfa7d","Type":"ContainerDied","Data":"e1d8e1211010cf0fc71792f2cd58c8a5a95de6b2311ec1d8e5dbaa16424ad724"} Nov 03 20:26:35.549025 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.548969 5111 scope.go:122] "RemoveContainer" containerID="dfa0cc2e93fd859038ce9bd941bc694d22671bbd001e35bf883153a042ff2b19" Nov 03 20:26:35.552057 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.551649 5111 generic.go:356] "Generic (PLEG): container finished" podID="5bfeea28-db58-4740-b089-9d8eb0dfece7" containerID="c02b619fc09eb6cedd9b240e63423df6af05c2c62823bde50046e8bd3b439f68" exitCode=0 Nov 03 20:26:35.552057 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.551895 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn" event={"ID":"5bfeea28-db58-4740-b089-9d8eb0dfece7","Type":"ContainerDied","Data":"c02b619fc09eb6cedd9b240e63423df6af05c2c62823bde50046e8bd3b439f68"} Nov 03 20:26:35.553991 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.553824 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" event={"ID":"38304073-d13f-4d9c-ac90-81ba88cb910b","Type":"ContainerDied","Data":"596c6bdf8fd1dd466fc5dc56cebb3c0da70d2099dc76a5eea0ecd18135ddb93a"} Nov 03 20:26:35.553991 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.553870 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="596c6bdf8fd1dd466fc5dc56cebb3c0da70d2099dc76a5eea0ecd18135ddb93a" Nov 03 20:26:35.553991 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.553933 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq" Nov 03 20:26:35.558084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.558061 5111 generic.go:356] "Generic (PLEG): container finished" podID="f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" containerID="8cbe909a2f6df9ace58ef88b130ed5d0c0d84709df84c735a7ec87bf0a7bf177" exitCode=0 Nov 03 20:26:35.558213 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.558091 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" event={"ID":"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f","Type":"ContainerDied","Data":"8cbe909a2f6df9ace58ef88b130ed5d0c0d84709df84c735a7ec87bf0a7bf177"} Nov 03 20:26:35.706694 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.706667 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:26:35.744440 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.744399 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-envoy\") pod \"027fe01b-c13b-4f05-babf-c440793dfa7d\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " Nov 03 20:26:35.744587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.744500 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-podinfo\") pod \"027fe01b-c13b-4f05-babf-c440793dfa7d\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " Nov 03 20:26:35.744729 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.744709 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-data\") pod \"027fe01b-c13b-4f05-babf-c440793dfa7d\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " Nov 03 20:26:35.744807 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.744773 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-token\" (UniqueName: \"kubernetes.io/projected/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-token\") pod \"027fe01b-c13b-4f05-babf-c440793dfa7d\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " Nov 03 20:26:35.744869 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.744833 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-workload-certs\") pod \"027fe01b-c13b-4f05-babf-c440793dfa7d\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " Nov 03 20:26:35.744920 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.744881 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027fe01b-c13b-4f05-babf-c440793dfa7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027fe01b-c13b-4f05-babf-c440793dfa7d-istiod-ca-cert\") pod \"027fe01b-c13b-4f05-babf-c440793dfa7d\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " Nov 03 20:26:35.745044 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.745026 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-workload-socket\") pod \"027fe01b-c13b-4f05-babf-c440793dfa7d\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " Nov 03 20:26:35.745110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.745070 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027fe01b-c13b-4f05-babf-c440793dfa7d-kube-api-access-z6w7r\" (UniqueName: \"kubernetes.io/projected/027fe01b-c13b-4f05-babf-c440793dfa7d-kube-api-access-z6w7r\") pod \"027fe01b-c13b-4f05-babf-c440793dfa7d\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " Nov 03 20:26:35.745175 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.745114 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-credential-socket\") pod \"027fe01b-c13b-4f05-babf-c440793dfa7d\" (UID: \"027fe01b-c13b-4f05-babf-c440793dfa7d\") " Nov 03 20:26:35.747154 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.747121 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-credential-socket" pod "027fe01b-c13b-4f05-babf-c440793dfa7d" (UID: "027fe01b-c13b-4f05-babf-c440793dfa7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:35.750189 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.750005 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-envoy" pod "027fe01b-c13b-4f05-babf-c440793dfa7d" (UID: "027fe01b-c13b-4f05-babf-c440793dfa7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:35.750427 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.750405 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027fe01b-c13b-4f05-babf-c440793dfa7d-istiod-ca-cert" pod "027fe01b-c13b-4f05-babf-c440793dfa7d" (UID: "027fe01b-c13b-4f05-babf-c440793dfa7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:35.751021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.750711 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-workload-socket" pod "027fe01b-c13b-4f05-babf-c440793dfa7d" (UID: "027fe01b-c13b-4f05-babf-c440793dfa7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:35.751216 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.751193 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-workload-certs" pod "027fe01b-c13b-4f05-babf-c440793dfa7d" (UID: "027fe01b-c13b-4f05-babf-c440793dfa7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:35.754555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.753833 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-podinfo" pod "027fe01b-c13b-4f05-babf-c440793dfa7d" (UID: "027fe01b-c13b-4f05-babf-c440793dfa7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:35.760865 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.760831 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-data" pod "027fe01b-c13b-4f05-babf-c440793dfa7d" (UID: "027fe01b-c13b-4f05-babf-c440793dfa7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:35.769869 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.769826 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027fe01b-c13b-4f05-babf-c440793dfa7d-kube-api-access-z6w7r" pod "027fe01b-c13b-4f05-babf-c440793dfa7d" (UID: "027fe01b-c13b-4f05-babf-c440793dfa7d"). InnerVolumeSpecName "kube-api-access-z6w7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:35.773966 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.771891 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" podUID="b8748fcc-2ada-4d73-afd8-4ebd35ee60f8" containerName="payload-processing" containerID="cri-o://2294862985ddf6c8b3afeddbef29014a6ea9cc838b227f5ab40cfc8a32198411" gracePeriod=25 Nov 03 20:26:35.781587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.780511 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-token" pod "027fe01b-c13b-4f05-babf-c440793dfa7d" (UID: "027fe01b-c13b-4f05-babf-c440793dfa7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:35.783238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.783209 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:26:35.785053 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.784993 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" podUID="f79a1068-5b22-4a40-8c5a-1565c2593369" containerName="payload-pre-processing" containerID="cri-o://39351c34f51037b782b36380bd64c1490c1be7277f0b7a4c4a1642248c5c7e39" gracePeriod=25 Nov 03 20:26:35.838557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.838115 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz"] Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.847987 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-credential-socket\") pod \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.848110 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-envoy\") pod \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.848156 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istiod-ca-cert\") pod \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.848183 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-workload-socket\") pod \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.848298 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-credential-socket" pod "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" (UID: "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.848380 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-podinfo\") pod \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.848410 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-token\" (UniqueName: \"kubernetes.io/projected/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-token\") pod \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.848496 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-data\") pod \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.848558 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-workload-certs\") pod \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.848588 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-kube-api-access-hhmwk\" (UniqueName: \"kubernetes.io/projected/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-kube-api-access-hhmwk\") pod \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\" (UID: \"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f\") " Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.849231 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.849254 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6w7r\" (UniqueName: \"kubernetes.io/projected/027fe01b-c13b-4f05-babf-c440793dfa7d-kube-api-access-z6w7r\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.849268 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.849282 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.849299 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.849313 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.849328 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.849341 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027fe01b-c13b-4f05-babf-c440793dfa7d-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.849355 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027fe01b-c13b-4f05-babf-c440793dfa7d-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.849368 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027fe01b-c13b-4f05-babf-c440793dfa7d-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.850275 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-workload-certs" pod "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" (UID: "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.850751 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istiod-ca-cert" pod "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" (UID: "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.851051 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-workload-socket" pod "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" (UID: "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:35.861903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.851262 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-data" pod "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" (UID: "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:35.868076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.864348 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-token" pod "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" (UID: "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:35.868076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.865352 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-podinfo" pod "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" (UID: "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:35.868076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.866973 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-envoy" pod "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" (UID: "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:35.868076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.867251 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-kube-api-access-hhmwk" pod "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" (UID: "f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f"). InnerVolumeSpecName "kube-api-access-hhmwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:35.951036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.950934 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.951036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.950970 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.951036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.950986 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhmwk\" (UniqueName: \"kubernetes.io/projected/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-kube-api-access-hhmwk\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.951036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.951000 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.951036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.951013 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.951036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.951026 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.951036 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.951038 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:35.951379 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:35.951051 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:36.066638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.066590 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" Nov 03 20:26:36.159370 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.159327 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8748fcc-2ada-4d73-afd8-4ebd35ee60f8-kube-api-access-92qvm\" (UniqueName: \"kubernetes.io/projected/b8748fcc-2ada-4d73-afd8-4ebd35ee60f8-kube-api-access-92qvm\") pod \"b8748fcc-2ada-4d73-afd8-4ebd35ee60f8\" (UID: \"b8748fcc-2ada-4d73-afd8-4ebd35ee60f8\") " Nov 03 20:26:36.160197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.160164 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8748fcc-2ada-4d73-afd8-4ebd35ee60f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8748fcc-2ada-4d73-afd8-4ebd35ee60f8-plugins-config-volume\") pod \"b8748fcc-2ada-4d73-afd8-4ebd35ee60f8\" (UID: \"b8748fcc-2ada-4d73-afd8-4ebd35ee60f8\") " Nov 03 20:26:36.161009 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.160983 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8748fcc-2ada-4d73-afd8-4ebd35ee60f8-plugins-config-volume" pod "b8748fcc-2ada-4d73-afd8-4ebd35ee60f8" (UID: "b8748fcc-2ada-4d73-afd8-4ebd35ee60f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:36.164262 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.164232 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8748fcc-2ada-4d73-afd8-4ebd35ee60f8-kube-api-access-92qvm" pod "b8748fcc-2ada-4d73-afd8-4ebd35ee60f8" (UID: "b8748fcc-2ada-4d73-afd8-4ebd35ee60f8"). InnerVolumeSpecName "kube-api-access-92qvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:36.194450 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.194408 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384309b2-620c-49b2-818f-793652ba6cc6" path="/var/lib/kubelet/pods/384309b2-620c-49b2-818f-793652ba6cc6/volumes" Nov 03 20:26:36.262335 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.262287 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92qvm\" (UniqueName: \"kubernetes.io/projected/b8748fcc-2ada-4d73-afd8-4ebd35ee60f8-kube-api-access-92qvm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:36.262335 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.262322 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8748fcc-2ada-4d73-afd8-4ebd35ee60f8-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:36.280376 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.280337 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk"] Nov 03 20:26:36.284500 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.284471 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-944e83-p9dgk"] Nov 03 20:26:36.571261 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.571229 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" Nov 03 20:26:36.571977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.571279 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl" event={"ID":"f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f","Type":"ContainerDied","Data":"4ec2250380d0a80fbbaca8e10e07fbb94f9fdcdaa7c03575d93520adfd82e204"} Nov 03 20:26:36.572176 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.572162 5111 scope.go:122] "RemoveContainer" containerID="8cbe909a2f6df9ace58ef88b130ed5d0c0d84709df84c735a7ec87bf0a7bf177" Nov 03 20:26:36.572739 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.572706 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" podUID="2c80ce62-7273-46ef-ba44-b9ced07ddfe2" containerName="payload-pre-processing" containerID="cri-o://e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8" gracePeriod=25 Nov 03 20:26:36.604817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.578957 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" event={"ID":"027fe01b-c13b-4f05-babf-c440793dfa7d","Type":"ContainerDied","Data":"5fe9c4b40527d7716e757518dd26c4a4e3600745649f91d17fc7979c7016afae"} Nov 03 20:26:36.604817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.579135 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx" Nov 03 20:26:36.604817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.600349 5111 generic.go:356] "Generic (PLEG): container finished" podID="b8748fcc-2ada-4d73-afd8-4ebd35ee60f8" containerID="2294862985ddf6c8b3afeddbef29014a6ea9cc838b227f5ab40cfc8a32198411" exitCode=0 Nov 03 20:26:36.604817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.603082 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" event={"ID":"b8748fcc-2ada-4d73-afd8-4ebd35ee60f8","Type":"ContainerDied","Data":"2294862985ddf6c8b3afeddbef29014a6ea9cc838b227f5ab40cfc8a32198411"} Nov 03 20:26:36.604817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.603244 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" event={"ID":"b8748fcc-2ada-4d73-afd8-4ebd35ee60f8","Type":"ContainerDied","Data":"28dab73a53b02188e66d8d453603b0f6d7ccd66070e14347c8c029085860713a"} Nov 03 20:26:36.613551 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.613502 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx" Nov 03 20:26:36.616799 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.616773 5111 scope.go:122] "RemoveContainer" containerID="e1d8e1211010cf0fc71792f2cd58c8a5a95de6b2311ec1d8e5dbaa16424ad724" Nov 03 20:26:36.683643 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.683284 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl"] Nov 03 20:26:36.689927 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.689893 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z"] Nov 03 20:26:36.699145 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.699106 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-htgdl"] Nov 03 20:26:36.705040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.705010 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ed729-zvn5z"] Nov 03 20:26:36.717641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.717581 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx"] Nov 03 20:26:36.723128 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.723060 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-h84jx"] Nov 03 20:26:36.724452 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.724167 5111 scope.go:122] "RemoveContainer" containerID="2294862985ddf6c8b3afeddbef29014a6ea9cc838b227f5ab40cfc8a32198411" Nov 03 20:26:36.731142 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.731101 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx"] Nov 03 20:26:36.738377 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.738355 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-27d3ad-5449b4d7d4-8qmlx"] Nov 03 20:26:36.747753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.747642 5111 scope.go:122] "RemoveContainer" containerID="2294862985ddf6c8b3afeddbef29014a6ea9cc838b227f5ab40cfc8a32198411" Nov 03 20:26:36.748048 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:36.747990 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2294862985ddf6c8b3afeddbef29014a6ea9cc838b227f5ab40cfc8a32198411\": container with ID starting with 2294862985ddf6c8b3afeddbef29014a6ea9cc838b227f5ab40cfc8a32198411 not found: ID does not exist" containerID="2294862985ddf6c8b3afeddbef29014a6ea9cc838b227f5ab40cfc8a32198411" Nov 03 20:26:36.748048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.748022 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2294862985ddf6c8b3afeddbef29014a6ea9cc838b227f5ab40cfc8a32198411"} err="failed to get container status \"2294862985ddf6c8b3afeddbef29014a6ea9cc838b227f5ab40cfc8a32198411\": rpc error: code = NotFound desc = could not find container \"2294862985ddf6c8b3afeddbef29014a6ea9cc838b227f5ab40cfc8a32198411\": container with ID starting with 2294862985ddf6c8b3afeddbef29014a6ea9cc838b227f5ab40cfc8a32198411 not found: ID does not exist" Nov 03 20:26:36.897555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.895474 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn" Nov 03 20:26:36.993164 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:36.993135 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" Nov 03 20:26:37.017738 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.017696 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bfeea28-db58-4740-b089-9d8eb0dfece7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfeea28-db58-4740-b089-9d8eb0dfece7-maas-api-xks-ca\") pod \"5bfeea28-db58-4740-b089-9d8eb0dfece7\" (UID: \"5bfeea28-db58-4740-b089-9d8eb0dfece7\") " Nov 03 20:26:37.017944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.017806 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfeea28-db58-4740-b089-9d8eb0dfece7-kube-api-access-gx5kj\" (UniqueName: \"kubernetes.io/projected/5bfeea28-db58-4740-b089-9d8eb0dfece7-kube-api-access-gx5kj\") pod \"5bfeea28-db58-4740-b089-9d8eb0dfece7\" (UID: \"5bfeea28-db58-4740-b089-9d8eb0dfece7\") " Nov 03 20:26:37.018018 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.017963 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bfeea28-db58-4740-b089-9d8eb0dfece7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfeea28-db58-4740-b089-9d8eb0dfece7-maas-api-openshift-ca\") pod \"5bfeea28-db58-4740-b089-9d8eb0dfece7\" (UID: \"5bfeea28-db58-4740-b089-9d8eb0dfece7\") " Nov 03 20:26:37.018983 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.018950 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfeea28-db58-4740-b089-9d8eb0dfece7-maas-api-openshift-ca" pod "5bfeea28-db58-4740-b089-9d8eb0dfece7" (UID: "5bfeea28-db58-4740-b089-9d8eb0dfece7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:37.021498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.021460 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bfeea28-db58-4740-b089-9d8eb0dfece7-maas-api-xks-ca" pod "5bfeea28-db58-4740-b089-9d8eb0dfece7" (UID: "5bfeea28-db58-4740-b089-9d8eb0dfece7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:37.021933 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.021898 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfeea28-db58-4740-b089-9d8eb0dfece7-kube-api-access-gx5kj" pod "5bfeea28-db58-4740-b089-9d8eb0dfece7" (UID: "5bfeea28-db58-4740-b089-9d8eb0dfece7"). InnerVolumeSpecName "kube-api-access-gx5kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:37.115944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.115852 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" podUID="f79a1068-5b22-4a40-8c5a-1565c2593369" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:26:37.119354 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.119310 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c80ce62-7273-46ef-ba44-b9ced07ddfe2-kube-api-access-gblml\" (UniqueName: \"kubernetes.io/projected/2c80ce62-7273-46ef-ba44-b9ced07ddfe2-kube-api-access-gblml\") pod \"2c80ce62-7273-46ef-ba44-b9ced07ddfe2\" (UID: \"2c80ce62-7273-46ef-ba44-b9ced07ddfe2\") " Nov 03 20:26:37.119510 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.119363 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c80ce62-7273-46ef-ba44-b9ced07ddfe2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c80ce62-7273-46ef-ba44-b9ced07ddfe2-plugins-config-volume\") pod \"2c80ce62-7273-46ef-ba44-b9ced07ddfe2\" (UID: \"2c80ce62-7273-46ef-ba44-b9ced07ddfe2\") " Nov 03 20:26:37.119868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.119840 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c80ce62-7273-46ef-ba44-b9ced07ddfe2-plugins-config-volume" pod "2c80ce62-7273-46ef-ba44-b9ced07ddfe2" (UID: "2c80ce62-7273-46ef-ba44-b9ced07ddfe2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:37.119983 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.119846 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfeea28-db58-4740-b089-9d8eb0dfece7-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:37.119983 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.119899 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfeea28-db58-4740-b089-9d8eb0dfece7-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:37.119983 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.119917 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx5kj\" (UniqueName: \"kubernetes.io/projected/5bfeea28-db58-4740-b089-9d8eb0dfece7-kube-api-access-gx5kj\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:37.121642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.121617 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c80ce62-7273-46ef-ba44-b9ced07ddfe2-kube-api-access-gblml" pod "2c80ce62-7273-46ef-ba44-b9ced07ddfe2" (UID: "2c80ce62-7273-46ef-ba44-b9ced07ddfe2"). InnerVolumeSpecName "kube-api-access-gblml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:37.236945 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.235835 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gblml\" (UniqueName: \"kubernetes.io/projected/2c80ce62-7273-46ef-ba44-b9ced07ddfe2-kube-api-access-gblml\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:37.236945 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.235876 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c80ce62-7273-46ef-ba44-b9ced07ddfe2-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:37.464677 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.464641 5111 scope.go:122] "RemoveContainer" containerID="600fa3efdfd9d2cd0821e42ac7eceac3e6d3f673b80ef9cf7e35e9dc1d3392a8" Nov 03 20:26:37.480296 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.480260 5111 scope.go:122] "RemoveContainer" containerID="5564cf8e282cd1db6698f4762e45bdc64f8d3b755235ea513403e31ad68ecb1b" Nov 03 20:26:37.609690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.609653 5111 scope.go:122] "RemoveContainer" containerID="cc28ec4020cd7a802f6297f6cd62c479bffcb36fdead0fd043f34c66bfef511d" Nov 03 20:26:37.633394 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.633359 5111 scope.go:122] "RemoveContainer" containerID="4864f0938dd6a7308d3f8d9601e79c36dba0d3ce9f8f151229aa6e5410381281" Nov 03 20:26:37.638039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.637965 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn" event={"ID":"5bfeea28-db58-4740-b089-9d8eb0dfece7","Type":"ContainerDied","Data":"4f54bb7ae84f0e6d8757e824e5e2dab2ce1c36b8084afef16e63d0f053c8e42e"} Nov 03 20:26:37.638039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.638019 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f54bb7ae84f0e6d8757e824e5e2dab2ce1c36b8084afef16e63d0f053c8e42e" Nov 03 20:26:37.638289 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.638273 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn" Nov 03 20:26:37.642213 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.642187 5111 generic.go:356] "Generic (PLEG): container finished" podID="2c80ce62-7273-46ef-ba44-b9ced07ddfe2" containerID="e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8" exitCode=0 Nov 03 20:26:37.645559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.642312 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" event={"ID":"2c80ce62-7273-46ef-ba44-b9ced07ddfe2","Type":"ContainerDied","Data":"e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8"} Nov 03 20:26:37.645559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.642357 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" event={"ID":"2c80ce62-7273-46ef-ba44-b9ced07ddfe2","Type":"ContainerDied","Data":"1efddefede5de86bb27930e18e1d00ecbf2f70762d127318b1070e2315bbd08e"} Nov 03 20:26:37.645559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.642376 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq" Nov 03 20:26:37.645559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.642385 5111 scope.go:122] "RemoveContainer" containerID="e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8" Nov 03 20:26:37.673090 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.672983 5111 scope.go:122] "RemoveContainer" containerID="b8b6149a8b76c68a93c283d2a15b48ed4cedf8520cf0d2fd8c9dd5dedf75cf66" Nov 03 20:26:37.682667 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.682642 5111 scope.go:122] "RemoveContainer" containerID="e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8" Nov 03 20:26:37.683007 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:37.682962 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8\": container with ID starting with e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8 not found: ID does not exist" containerID="e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8" Nov 03 20:26:37.683095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.683019 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8"} err="failed to get container status \"e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8\": rpc error: code = NotFound desc = could not find container \"e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8\": container with ID starting with e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8 not found: ID does not exist" Nov 03 20:26:37.736864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.736766 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq"] Nov 03 20:26:37.743314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.743283 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-944e83-db6cbc767-62qqq"] Nov 03 20:26:37.755206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.755057 5111 scope.go:122] "RemoveContainer" containerID="2910d77601ca1a49b55f41e2377dbe1f821bd797a64ada2d3f8584495dd02752" Nov 03 20:26:37.800606 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.800569 5111 scope.go:122] "RemoveContainer" containerID="e48a351c8b04aa9b84f50c99197d767c5663c641e0b0a4cbab4a1bf380f92d5b" Nov 03 20:26:37.818424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.818342 5111 scope.go:122] "RemoveContainer" containerID="4c5a39801ee00b39bbcd95b3344fe6d6f501b98709ed0ad1a88a6499705305da" Nov 03 20:26:37.835912 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.835618 5111 scope.go:122] "RemoveContainer" containerID="596b6381aaacf97e444281b1527c932ae1002e90714b8e06ef8d91b1e2b30bc0" Nov 03 20:26:37.849892 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:37.848172 5111 scope.go:122] "RemoveContainer" containerID="e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8" Nov 03 20:26:37.849892 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:37.848526 5111 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8\": rpc error: code = NotFound desc = could not find container \"e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8\": container with ID starting with e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8 not found: ID does not exist" containerID="e1f43db1d05a0ef04237223db53c21cb002d91c23049d154d3148bbcd4aecae8" Nov 03 20:26:38.194557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:38.194200 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" path="/var/lib/kubelet/pods/027fe01b-c13b-4f05-babf-c440793dfa7d/volumes" Nov 03 20:26:38.196247 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:38.195478 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c80ce62-7273-46ef-ba44-b9ced07ddfe2" path="/var/lib/kubelet/pods/2c80ce62-7273-46ef-ba44-b9ced07ddfe2/volumes" Nov 03 20:26:38.196247 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:38.196212 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="550ca9cd-5b2e-435d-a9d9-958f65b7e5a0" path="/var/lib/kubelet/pods/550ca9cd-5b2e-435d-a9d9-958f65b7e5a0/volumes" Nov 03 20:26:38.198082 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:38.197242 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78cdee3f-8ad2-4767-bf61-9c8eae1b76cc" path="/var/lib/kubelet/pods/78cdee3f-8ad2-4767-bf61-9c8eae1b76cc/volumes" Nov 03 20:26:38.198082 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:38.197784 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8748fcc-2ada-4d73-afd8-4ebd35ee60f8" path="/var/lib/kubelet/pods/b8748fcc-2ada-4d73-afd8-4ebd35ee60f8/volumes" Nov 03 20:26:38.198828 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:38.198791 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" path="/var/lib/kubelet/pods/f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f/volumes" Nov 03 20:26:41.143752 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:41.143713 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4"] Nov 03 20:26:41.157856 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:41.157815 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2"] Nov 03 20:26:42.524138 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:42.524101 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn"] Nov 03 20:26:42.524931 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:42.524897 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" podUID="d5b33145-a82b-4519-ab6a-c7ce0248992f" containerName="istio-proxy" containerID="cri-o://ad67ec99016939ed0b81e2b594857c37cd515de92a6bfdd357f477b7662cbc9a" gracePeriod=30 Nov 03 20:26:43.216413 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.216372 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j"] Nov 03 20:26:43.217270 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217250 5111 cpu_manager.go:414] "containerMap: removing container" podUID="f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" containerName="istio-proxy" Nov 03 20:26:43.217318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217274 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" containerName="istio-proxy" Nov 03 20:26:43.217318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217296 5111 cpu_manager.go:414] "containerMap: removing container" podUID="38304073-d13f-4d9c-ac90-81ba88cb910b" containerName="revoke-keys" Nov 03 20:26:43.217318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217301 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38304073-d13f-4d9c-ac90-81ba88cb910b" containerName="revoke-keys" Nov 03 20:26:43.217318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217309 5111 cpu_manager.go:414] "containerMap: removing container" podUID="5bfeea28-db58-4740-b089-9d8eb0dfece7" containerName="revoke-keys" Nov 03 20:26:43.217318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217314 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfeea28-db58-4740-b089-9d8eb0dfece7" containerName="revoke-keys" Nov 03 20:26:43.217478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217330 5111 cpu_manager.go:414] "containerMap: removing container" podUID="2c80ce62-7273-46ef-ba44-b9ced07ddfe2" containerName="payload-pre-processing" Nov 03 20:26:43.217478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217334 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c80ce62-7273-46ef-ba44-b9ced07ddfe2" containerName="payload-pre-processing" Nov 03 20:26:43.217478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217341 5111 cpu_manager.go:414] "containerMap: removing container" podUID="b8748fcc-2ada-4d73-afd8-4ebd35ee60f8" containerName="payload-processing" Nov 03 20:26:43.217478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217346 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8748fcc-2ada-4d73-afd8-4ebd35ee60f8" containerName="payload-processing" Nov 03 20:26:43.217478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217356 5111 cpu_manager.go:414] "containerMap: removing container" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" Nov 03 20:26:43.217478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217361 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" Nov 03 20:26:43.217478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217374 5111 cpu_manager.go:414] "containerMap: removing container" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" Nov 03 20:26:43.217478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217378 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" Nov 03 20:26:43.217744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217489 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" Nov 03 20:26:43.217744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217503 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="f47c1ee1-ba29-43e7-a737-5f2a6a8fa56f" containerName="istio-proxy" Nov 03 20:26:43.217744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217511 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="38304073-d13f-4d9c-ac90-81ba88cb910b" containerName="revoke-keys" Nov 03 20:26:43.217744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217518 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfeea28-db58-4740-b089-9d8eb0dfece7" containerName="revoke-keys" Nov 03 20:26:43.217744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217526 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c80ce62-7273-46ef-ba44-b9ced07ddfe2" containerName="payload-pre-processing" Nov 03 20:26:43.217744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.217546 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8748fcc-2ada-4d73-afd8-4ebd35ee60f8" containerName="payload-processing" Nov 03 20:26:43.221157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.221138 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" Nov 03 20:26:43.226646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.226599 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:26:43.226817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.226799 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:26:43.231389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.231360 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j"] Nov 03 20:26:43.306228 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.306177 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lshpk\" (UniqueName: \"kubernetes.io/projected/e62ed539-4a84-4b9e-bd92-d93c67e928b4-kube-api-access-lshpk\") pod \"maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j\" (UID: \"e62ed539-4a84-4b9e-bd92-d93c67e928b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" Nov 03 20:26:43.306395 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.306360 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62ed539-4a84-4b9e-bd92-d93c67e928b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j\" (UID: \"e62ed539-4a84-4b9e-bd92-d93c67e928b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" Nov 03 20:26:43.306480 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.306405 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62ed539-4a84-4b9e-bd92-d93c67e928b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j\" (UID: \"e62ed539-4a84-4b9e-bd92-d93c67e928b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" Nov 03 20:26:43.313906 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.313875 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:26:43.314100 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.313974 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:26:43.408081 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.408041 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62ed539-4a84-4b9e-bd92-d93c67e928b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j\" (UID: \"e62ed539-4a84-4b9e-bd92-d93c67e928b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" Nov 03 20:26:43.408305 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.408269 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshpk\" (UniqueName: \"kubernetes.io/projected/e62ed539-4a84-4b9e-bd92-d93c67e928b4-kube-api-access-lshpk\") pod \"maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j\" (UID: \"e62ed539-4a84-4b9e-bd92-d93c67e928b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" Nov 03 20:26:43.408408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.408391 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62ed539-4a84-4b9e-bd92-d93c67e928b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j\" (UID: \"e62ed539-4a84-4b9e-bd92-d93c67e928b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" Nov 03 20:26:43.409079 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.409058 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62ed539-4a84-4b9e-bd92-d93c67e928b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j\" (UID: \"e62ed539-4a84-4b9e-bd92-d93c67e928b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" Nov 03 20:26:43.410455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.410438 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62ed539-4a84-4b9e-bd92-d93c67e928b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j\" (UID: \"e62ed539-4a84-4b9e-bd92-d93c67e928b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" Nov 03 20:26:43.416963 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.416940 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshpk\" (UniqueName: \"kubernetes.io/projected/e62ed539-4a84-4b9e-bd92-d93c67e928b4-kube-api-access-lshpk\") pod \"maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j\" (UID: \"e62ed539-4a84-4b9e-bd92-d93c67e928b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" Nov 03 20:26:43.547121 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.547029 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" Nov 03 20:26:43.852977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:43.852939 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j"] Nov 03 20:26:43.853220 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:26:43.853193 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62ed539_4a84_4b9e_bd92_d93c67e928b4.slice/crio-02d73911fab0eb72facd98e1e7a0754c15d355c8113aa6de2a52c09291ba0ece WatchSource:0}: Error finding container 02d73911fab0eb72facd98e1e7a0754c15d355c8113aa6de2a52c09291ba0ece: Status 404 returned error can't find the container with id 02d73911fab0eb72facd98e1e7a0754c15d355c8113aa6de2a52c09291ba0ece Nov 03 20:26:44.053399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:44.053365 5111 patch_prober.go:29] interesting pod/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:26:44.053554 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:44.053424 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" podUID="d5b33145-a82b-4519-ab6a-c7ce0248992f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:26:44.363352 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:44.363319 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc"] Nov 03 20:26:44.363643 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:44.363593 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" podUID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" containerName="istio-proxy" containerID="cri-o://bde78d8e8312c48a3f78f696d06ee9bf6e7f13ab37f1f5dc1b5824ed2a6e13e0" gracePeriod=30 Nov 03 20:26:44.728491 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:44.728450 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" event={"ID":"e62ed539-4a84-4b9e-bd92-d93c67e928b4","Type":"ContainerStarted","Data":"00d05397f20e9bd989209a3ff9b407868b5bb14d0698f78bdb2c7a604c5c9274"} Nov 03 20:26:44.729013 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:44.728512 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" event={"ID":"e62ed539-4a84-4b9e-bd92-d93c67e928b4","Type":"ContainerStarted","Data":"02d73911fab0eb72facd98e1e7a0754c15d355c8113aa6de2a52c09291ba0ece"} Nov 03 20:26:44.747744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:44.747688 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" podStartSLOduration=1.74766918 podStartE2EDuration="1.74766918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:44.745845772 +0000 UTC m=+2834.708224211" watchObservedRunningTime="2026-11-03 20:26:44.74766918 +0000 UTC m=+2834.710047625" Nov 03 20:26:45.010953 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.010876 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr"] Nov 03 20:26:45.013423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.011878 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="027fe01b-c13b-4f05-babf-c440793dfa7d" containerName="istio-proxy" Nov 03 20:26:45.016320 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.016296 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" Nov 03 20:26:45.062930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.062827 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr"] Nov 03 20:26:45.075930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.075891 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss"] Nov 03 20:26:45.076325 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.076295 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" podUID="dd161e75-9c5f-4e1c-8a76-5a0171162c42" containerName="istio-proxy" containerID="cri-o://be379da620d03d13c5d4c536bd4776b855f69a63874df9b83a6ce0c0066b5a82" gracePeriod=30 Nov 03 20:26:45.127952 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.127911 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:26:45.128117 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.127970 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" podUID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:26:45.161630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.161586 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d4gd\" (UniqueName: \"kubernetes.io/projected/f5d2febc-f78a-4494-b202-18f8fdfe80cb-kube-api-access-6d4gd\") pod \"maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr\" (UID: \"f5d2febc-f78a-4494-b202-18f8fdfe80cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" Nov 03 20:26:45.161852 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.161662 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d2febc-f78a-4494-b202-18f8fdfe80cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr\" (UID: \"f5d2febc-f78a-4494-b202-18f8fdfe80cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" Nov 03 20:26:45.161852 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.161816 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d2febc-f78a-4494-b202-18f8fdfe80cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr\" (UID: \"f5d2febc-f78a-4494-b202-18f8fdfe80cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" Nov 03 20:26:45.263658 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.263555 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d2febc-f78a-4494-b202-18f8fdfe80cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr\" (UID: \"f5d2febc-f78a-4494-b202-18f8fdfe80cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" Nov 03 20:26:45.263854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.263783 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d2febc-f78a-4494-b202-18f8fdfe80cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr\" (UID: \"f5d2febc-f78a-4494-b202-18f8fdfe80cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" Nov 03 20:26:45.264110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.264079 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d4gd\" (UniqueName: \"kubernetes.io/projected/f5d2febc-f78a-4494-b202-18f8fdfe80cb-kube-api-access-6d4gd\") pod \"maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr\" (UID: \"f5d2febc-f78a-4494-b202-18f8fdfe80cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" Nov 03 20:26:45.264501 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.264479 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d2febc-f78a-4494-b202-18f8fdfe80cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr\" (UID: \"f5d2febc-f78a-4494-b202-18f8fdfe80cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" Nov 03 20:26:45.266418 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.266396 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d2febc-f78a-4494-b202-18f8fdfe80cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr\" (UID: \"f5d2febc-f78a-4494-b202-18f8fdfe80cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" Nov 03 20:26:45.275170 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.275143 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d4gd\" (UniqueName: \"kubernetes.io/projected/f5d2febc-f78a-4494-b202-18f8fdfe80cb-kube-api-access-6d4gd\") pod \"maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr\" (UID: \"f5d2febc-f78a-4494-b202-18f8fdfe80cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" Nov 03 20:26:45.383627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.383591 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" Nov 03 20:26:45.672457 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.672423 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr"] Nov 03 20:26:45.673199 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:26:45.673167 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d2febc_f78a_4494_b202_18f8fdfe80cb.slice/crio-1d608794c9f3a751a2f4808f9678d1f277769c4a066b18c6ce6f7f80786580ca WatchSource:0}: Error finding container 1d608794c9f3a751a2f4808f9678d1f277769c4a066b18c6ce6f7f80786580ca: Status 404 returned error can't find the container with id 1d608794c9f3a751a2f4808f9678d1f277769c4a066b18c6ce6f7f80786580ca Nov 03 20:26:45.737031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.736996 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" event={"ID":"f5d2febc-f78a-4494-b202-18f8fdfe80cb","Type":"ContainerStarted","Data":"1d608794c9f3a751a2f4808f9678d1f277769c4a066b18c6ce6f7f80786580ca"} Nov 03 20:26:45.738590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.738521 5111 generic.go:356] "Generic (PLEG): container finished" podID="e62ed539-4a84-4b9e-bd92-d93c67e928b4" containerID="00d05397f20e9bd989209a3ff9b407868b5bb14d0698f78bdb2c7a604c5c9274" exitCode=0 Nov 03 20:26:45.738724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:45.738557 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" event={"ID":"e62ed539-4a84-4b9e-bd92-d93c67e928b4","Type":"ContainerDied","Data":"00d05397f20e9bd989209a3ff9b407868b5bb14d0698f78bdb2c7a604c5c9274"} Nov 03 20:26:46.144604 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.144541 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" podUID="9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4" containerName="payload-processing" containerID="cri-o://1853bc768b19ef71caf8e5968373d6c4b909c1d2edd6a1acdd4ce2b101eac0b6" gracePeriod=25 Nov 03 20:26:46.158280 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.158238 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" podUID="267cf330-341c-40a3-9b8e-695d9273786a" containerName="payload-pre-processing" containerID="cri-o://c6bb762c312cf8de715601a690653b875f152059577fa67f341b4ad7d182a7be" gracePeriod=25 Nov 03 20:26:46.431040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.431002 5111 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:26:46.431200 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.431070 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" podUID="dd161e75-9c5f-4e1c-8a76-5a0171162c42" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:26:46.450264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.450234 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" Nov 03 20:26:46.483465 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.483435 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4-kube-api-access-zqc85\" (UniqueName: \"kubernetes.io/projected/9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4-kube-api-access-zqc85\") pod \"9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4\" (UID: \"9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4\") " Nov 03 20:26:46.483871 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.483630 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4-plugins-config-volume\") pod \"9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4\" (UID: \"9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4\") " Nov 03 20:26:46.484123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.484094 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4-plugins-config-volume" pod "9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4" (UID: "9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:46.484401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.484381 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:46.486219 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.486190 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4-kube-api-access-zqc85" pod "9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4" (UID: "9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4"). InnerVolumeSpecName "kube-api-access-zqc85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:46.512121 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.512092 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" Nov 03 20:26:46.585012 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.584918 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267cf330-341c-40a3-9b8e-695d9273786a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267cf330-341c-40a3-9b8e-695d9273786a-plugins-config-volume\") pod \"267cf330-341c-40a3-9b8e-695d9273786a\" (UID: \"267cf330-341c-40a3-9b8e-695d9273786a\") " Nov 03 20:26:46.585012 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.585002 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267cf330-341c-40a3-9b8e-695d9273786a-kube-api-access-8mx2q\" (UniqueName: \"kubernetes.io/projected/267cf330-341c-40a3-9b8e-695d9273786a-kube-api-access-8mx2q\") pod \"267cf330-341c-40a3-9b8e-695d9273786a\" (UID: \"267cf330-341c-40a3-9b8e-695d9273786a\") " Nov 03 20:26:46.585332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.585320 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqc85\" (UniqueName: \"kubernetes.io/projected/9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4-kube-api-access-zqc85\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:46.585396 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.585371 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267cf330-341c-40a3-9b8e-695d9273786a-plugins-config-volume" pod "267cf330-341c-40a3-9b8e-695d9273786a" (UID: "267cf330-341c-40a3-9b8e-695d9273786a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:46.587163 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.587139 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267cf330-341c-40a3-9b8e-695d9273786a-kube-api-access-8mx2q" pod "267cf330-341c-40a3-9b8e-695d9273786a" (UID: "267cf330-341c-40a3-9b8e-695d9273786a"). InnerVolumeSpecName "kube-api-access-8mx2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:46.686344 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.686313 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267cf330-341c-40a3-9b8e-695d9273786a-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:46.686344 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.686346 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mx2q\" (UniqueName: \"kubernetes.io/projected/267cf330-341c-40a3-9b8e-695d9273786a-kube-api-access-8mx2q\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:46.747924 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.747886 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" event={"ID":"f5d2febc-f78a-4494-b202-18f8fdfe80cb","Type":"ContainerStarted","Data":"fac5c40e8439fed67cce1c68d3c2d92691558723c66cf6b59f9494c8bf934012"} Nov 03 20:26:46.749584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.749554 5111 generic.go:356] "Generic (PLEG): container finished" podID="267cf330-341c-40a3-9b8e-695d9273786a" containerID="c6bb762c312cf8de715601a690653b875f152059577fa67f341b4ad7d182a7be" exitCode=0 Nov 03 20:26:46.749702 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.749591 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" event={"ID":"267cf330-341c-40a3-9b8e-695d9273786a","Type":"ContainerDied","Data":"c6bb762c312cf8de715601a690653b875f152059577fa67f341b4ad7d182a7be"} Nov 03 20:26:46.749702 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.749609 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" Nov 03 20:26:46.749702 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.749643 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2" event={"ID":"267cf330-341c-40a3-9b8e-695d9273786a","Type":"ContainerDied","Data":"af74551275fce5acf6b175627cb657f1c0062fd4e89dd06478ef4b6afddce388"} Nov 03 20:26:46.749702 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.749669 5111 scope.go:122] "RemoveContainer" containerID="c6bb762c312cf8de715601a690653b875f152059577fa67f341b4ad7d182a7be" Nov 03 20:26:46.751313 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.751288 5111 generic.go:356] "Generic (PLEG): container finished" podID="9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4" containerID="1853bc768b19ef71caf8e5968373d6c4b909c1d2edd6a1acdd4ce2b101eac0b6" exitCode=0 Nov 03 20:26:46.751433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.751352 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" Nov 03 20:26:46.751433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.751408 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" event={"ID":"9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4","Type":"ContainerDied","Data":"1853bc768b19ef71caf8e5968373d6c4b909c1d2edd6a1acdd4ce2b101eac0b6"} Nov 03 20:26:46.751604 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.751449 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4" event={"ID":"9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4","Type":"ContainerDied","Data":"fb712a19ab08243c50e4f2ceb9a1c660bf776371a39cb1e1f4161be03fe22393"} Nov 03 20:26:46.772207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.772154 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" podStartSLOduration=2.772139268 podStartE2EDuration="2.772139268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:46.769905241 +0000 UTC m=+2836.732283679" watchObservedRunningTime="2026-11-03 20:26:46.772139268 +0000 UTC m=+2836.734517705" Nov 03 20:26:46.793687 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.793666 5111 scope.go:122] "RemoveContainer" containerID="c6bb762c312cf8de715601a690653b875f152059577fa67f341b4ad7d182a7be" Nov 03 20:26:46.794023 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:46.793990 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6bb762c312cf8de715601a690653b875f152059577fa67f341b4ad7d182a7be\": container with ID starting with c6bb762c312cf8de715601a690653b875f152059577fa67f341b4ad7d182a7be not found: ID does not exist" containerID="c6bb762c312cf8de715601a690653b875f152059577fa67f341b4ad7d182a7be" Nov 03 20:26:46.794130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.794030 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6bb762c312cf8de715601a690653b875f152059577fa67f341b4ad7d182a7be"} err="failed to get container status \"c6bb762c312cf8de715601a690653b875f152059577fa67f341b4ad7d182a7be\": rpc error: code = NotFound desc = could not find container \"c6bb762c312cf8de715601a690653b875f152059577fa67f341b4ad7d182a7be\": container with ID starting with c6bb762c312cf8de715601a690653b875f152059577fa67f341b4ad7d182a7be not found: ID does not exist" Nov 03 20:26:46.794130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.794047 5111 scope.go:122] "RemoveContainer" containerID="1853bc768b19ef71caf8e5968373d6c4b909c1d2edd6a1acdd4ce2b101eac0b6" Nov 03 20:26:46.804833 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.804809 5111 scope.go:122] "RemoveContainer" containerID="1853bc768b19ef71caf8e5968373d6c4b909c1d2edd6a1acdd4ce2b101eac0b6" Nov 03 20:26:46.805261 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:46.805140 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1853bc768b19ef71caf8e5968373d6c4b909c1d2edd6a1acdd4ce2b101eac0b6\": container with ID starting with 1853bc768b19ef71caf8e5968373d6c4b909c1d2edd6a1acdd4ce2b101eac0b6 not found: ID does not exist" containerID="1853bc768b19ef71caf8e5968373d6c4b909c1d2edd6a1acdd4ce2b101eac0b6" Nov 03 20:26:46.805261 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.805169 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1853bc768b19ef71caf8e5968373d6c4b909c1d2edd6a1acdd4ce2b101eac0b6"} err="failed to get container status \"1853bc768b19ef71caf8e5968373d6c4b909c1d2edd6a1acdd4ce2b101eac0b6\": rpc error: code = NotFound desc = could not find container \"1853bc768b19ef71caf8e5968373d6c4b909c1d2edd6a1acdd4ce2b101eac0b6\": container with ID starting with 1853bc768b19ef71caf8e5968373d6c4b909c1d2edd6a1acdd4ce2b101eac0b6 not found: ID does not exist" Nov 03 20:26:46.808985 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.808962 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2"] Nov 03 20:26:46.812814 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.812791 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-120fe3-59798bf69f-c6dx2"] Nov 03 20:26:46.846905 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.843920 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4"] Nov 03 20:26:46.856610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.853377 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-120fe3-564ff75dfb-hscr4"] Nov 03 20:26:46.915046 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.915015 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" Nov 03 20:26:46.989465 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.989435 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62ed539-4a84-4b9e-bd92-d93c67e928b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62ed539-4a84-4b9e-bd92-d93c67e928b4-maas-api-openshift-ca\") pod \"e62ed539-4a84-4b9e-bd92-d93c67e928b4\" (UID: \"e62ed539-4a84-4b9e-bd92-d93c67e928b4\") " Nov 03 20:26:46.989648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.989564 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62ed539-4a84-4b9e-bd92-d93c67e928b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62ed539-4a84-4b9e-bd92-d93c67e928b4-maas-api-xks-ca\") pod \"e62ed539-4a84-4b9e-bd92-d93c67e928b4\" (UID: \"e62ed539-4a84-4b9e-bd92-d93c67e928b4\") " Nov 03 20:26:46.989648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.989587 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62ed539-4a84-4b9e-bd92-d93c67e928b4-kube-api-access-lshpk\" (UniqueName: \"kubernetes.io/projected/e62ed539-4a84-4b9e-bd92-d93c67e928b4-kube-api-access-lshpk\") pod \"e62ed539-4a84-4b9e-bd92-d93c67e928b4\" (UID: \"e62ed539-4a84-4b9e-bd92-d93c67e928b4\") " Nov 03 20:26:46.989854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.989830 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62ed539-4a84-4b9e-bd92-d93c67e928b4-maas-api-openshift-ca" pod "e62ed539-4a84-4b9e-bd92-d93c67e928b4" (UID: "e62ed539-4a84-4b9e-bd92-d93c67e928b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:46.990073 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.990056 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62ed539-4a84-4b9e-bd92-d93c67e928b4-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:46.991827 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.991802 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62ed539-4a84-4b9e-bd92-d93c67e928b4-maas-api-xks-ca" pod "e62ed539-4a84-4b9e-bd92-d93c67e928b4" (UID: "e62ed539-4a84-4b9e-bd92-d93c67e928b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:46.991926 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:46.991898 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62ed539-4a84-4b9e-bd92-d93c67e928b4-kube-api-access-lshpk" pod "e62ed539-4a84-4b9e-bd92-d93c67e928b4" (UID: "e62ed539-4a84-4b9e-bd92-d93c67e928b4"). InnerVolumeSpecName "kube-api-access-lshpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:47.090922 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.090888 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62ed539-4a84-4b9e-bd92-d93c67e928b4-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:47.090922 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.090921 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lshpk\" (UniqueName: \"kubernetes.io/projected/e62ed539-4a84-4b9e-bd92-d93c67e928b4-kube-api-access-lshpk\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:47.116246 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.116155 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" podUID="f79a1068-5b22-4a40-8c5a-1565c2593369" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:26:47.313579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.313542 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7"] Nov 03 20:26:47.314134 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.314120 5111 cpu_manager.go:414] "containerMap: removing container" podUID="e62ed539-4a84-4b9e-bd92-d93c67e928b4" containerName="revoke-keys" Nov 03 20:26:47.314180 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.314135 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62ed539-4a84-4b9e-bd92-d93c67e928b4" containerName="revoke-keys" Nov 03 20:26:47.314180 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.314160 5111 cpu_manager.go:414] "containerMap: removing container" podUID="267cf330-341c-40a3-9b8e-695d9273786a" containerName="payload-pre-processing" Nov 03 20:26:47.314180 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.314166 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267cf330-341c-40a3-9b8e-695d9273786a" containerName="payload-pre-processing" Nov 03 20:26:47.314286 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.314189 5111 cpu_manager.go:414] "containerMap: removing container" podUID="9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4" containerName="payload-processing" Nov 03 20:26:47.314286 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.314207 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4" containerName="payload-processing" Nov 03 20:26:47.314353 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.314296 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="267cf330-341c-40a3-9b8e-695d9273786a" containerName="payload-pre-processing" Nov 03 20:26:47.314353 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.314309 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4" containerName="payload-processing" Nov 03 20:26:47.314353 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.314319 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62ed539-4a84-4b9e-bd92-d93c67e928b4" containerName="revoke-keys" Nov 03 20:26:47.318232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.318210 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" Nov 03 20:26:47.328170 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.327909 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7"] Nov 03 20:26:47.334199 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.334172 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn"] Nov 03 20:26:47.365468 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.365438 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn"] Nov 03 20:26:47.365838 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.365812 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" Nov 03 20:26:47.395088 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.395049 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad12f36-4946-4027-a489-7a7f4b6b205b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e03b1fa6-68vqn\" (UID: \"fad12f36-4946-4027-a489-7a7f4b6b205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" Nov 03 20:26:47.395238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.395149 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tc8h\" (UniqueName: \"kubernetes.io/projected/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-kube-api-access-2tc8h\") pod \"maas-api-revoke-sub-85f57365-j4tb7\" (UID: \"5dd1beb8-21b5-49f9-a228-e2ddf280fc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" Nov 03 20:26:47.395238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.395230 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85f57365-j4tb7\" (UID: \"5dd1beb8-21b5-49f9-a228-e2ddf280fc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" Nov 03 20:26:47.395332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.395284 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad12f36-4946-4027-a489-7a7f4b6b205b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e03b1fa6-68vqn\" (UID: \"fad12f36-4946-4027-a489-7a7f4b6b205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" Nov 03 20:26:47.395391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.395375 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mctqv\" (UniqueName: \"kubernetes.io/projected/fad12f36-4946-4027-a489-7a7f4b6b205b-kube-api-access-mctqv\") pod \"maas-api-revoke-sub-e03b1fa6-68vqn\" (UID: \"fad12f36-4946-4027-a489-7a7f4b6b205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" Nov 03 20:26:47.395453 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.395440 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85f57365-j4tb7\" (UID: \"5dd1beb8-21b5-49f9-a228-e2ddf280fc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" Nov 03 20:26:47.496894 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.496856 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85f57365-j4tb7\" (UID: \"5dd1beb8-21b5-49f9-a228-e2ddf280fc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" Nov 03 20:26:47.497052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.497007 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad12f36-4946-4027-a489-7a7f4b6b205b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e03b1fa6-68vqn\" (UID: \"fad12f36-4946-4027-a489-7a7f4b6b205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" Nov 03 20:26:47.497127 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.497079 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tc8h\" (UniqueName: \"kubernetes.io/projected/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-kube-api-access-2tc8h\") pod \"maas-api-revoke-sub-85f57365-j4tb7\" (UID: \"5dd1beb8-21b5-49f9-a228-e2ddf280fc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" Nov 03 20:26:47.497187 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.497150 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85f57365-j4tb7\" (UID: \"5dd1beb8-21b5-49f9-a228-e2ddf280fc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" Nov 03 20:26:47.497238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.497196 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad12f36-4946-4027-a489-7a7f4b6b205b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e03b1fa6-68vqn\" (UID: \"fad12f36-4946-4027-a489-7a7f4b6b205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" Nov 03 20:26:47.497298 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.497281 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mctqv\" (UniqueName: \"kubernetes.io/projected/fad12f36-4946-4027-a489-7a7f4b6b205b-kube-api-access-mctqv\") pod \"maas-api-revoke-sub-e03b1fa6-68vqn\" (UID: \"fad12f36-4946-4027-a489-7a7f4b6b205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" Nov 03 20:26:47.497829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.497798 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad12f36-4946-4027-a489-7a7f4b6b205b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e03b1fa6-68vqn\" (UID: \"fad12f36-4946-4027-a489-7a7f4b6b205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" Nov 03 20:26:47.497829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.497815 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85f57365-j4tb7\" (UID: \"5dd1beb8-21b5-49f9-a228-e2ddf280fc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" Nov 03 20:26:47.499424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.499400 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85f57365-j4tb7\" (UID: \"5dd1beb8-21b5-49f9-a228-e2ddf280fc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" Nov 03 20:26:47.499625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.499607 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad12f36-4946-4027-a489-7a7f4b6b205b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e03b1fa6-68vqn\" (UID: \"fad12f36-4946-4027-a489-7a7f4b6b205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" Nov 03 20:26:47.509000 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.508977 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mctqv\" (UniqueName: \"kubernetes.io/projected/fad12f36-4946-4027-a489-7a7f4b6b205b-kube-api-access-mctqv\") pod \"maas-api-revoke-sub-e03b1fa6-68vqn\" (UID: \"fad12f36-4946-4027-a489-7a7f4b6b205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" Nov 03 20:26:47.511391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.511370 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tc8h\" (UniqueName: \"kubernetes.io/projected/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-kube-api-access-2tc8h\") pod \"maas-api-revoke-sub-85f57365-j4tb7\" (UID: \"5dd1beb8-21b5-49f9-a228-e2ddf280fc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" Nov 03 20:26:47.666204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.666165 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" Nov 03 20:26:47.682810 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.682565 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" Nov 03 20:26:47.706692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.706662 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn"] Nov 03 20:26:47.712278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.712245 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-456f55-557sn"] Nov 03 20:26:47.792803 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.792767 5111 generic.go:356] "Generic (PLEG): container finished" podID="f5d2febc-f78a-4494-b202-18f8fdfe80cb" containerID="fac5c40e8439fed67cce1c68d3c2d92691558723c66cf6b59f9494c8bf934012" exitCode=0 Nov 03 20:26:47.793111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.792896 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" event={"ID":"f5d2febc-f78a-4494-b202-18f8fdfe80cb","Type":"ContainerDied","Data":"fac5c40e8439fed67cce1c68d3c2d92691558723c66cf6b59f9494c8bf934012"} Nov 03 20:26:47.795100 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.795078 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" Nov 03 20:26:47.797838 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.797806 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j" event={"ID":"e62ed539-4a84-4b9e-bd92-d93c67e928b4","Type":"ContainerDied","Data":"02d73911fab0eb72facd98e1e7a0754c15d355c8113aa6de2a52c09291ba0ece"} Nov 03 20:26:47.797963 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.797866 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02d73911fab0eb72facd98e1e7a0754c15d355c8113aa6de2a52c09291ba0ece" Nov 03 20:26:47.805150 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.805123 5111 generic.go:356] "Generic (PLEG): container finished" podID="d5b33145-a82b-4519-ab6a-c7ce0248992f" containerID="ad67ec99016939ed0b81e2b594857c37cd515de92a6bfdd357f477b7662cbc9a" exitCode=0 Nov 03 20:26:47.805253 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.805200 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" event={"ID":"d5b33145-a82b-4519-ab6a-c7ce0248992f","Type":"ContainerDied","Data":"ad67ec99016939ed0b81e2b594857c37cd515de92a6bfdd357f477b7662cbc9a"} Nov 03 20:26:47.850797 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.850764 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:26:47.906869 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.906772 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-credential-socket\") pod \"d5b33145-a82b-4519-ab6a-c7ce0248992f\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " Nov 03 20:26:47.907150 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.907115 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b33145-a82b-4519-ab6a-c7ce0248992f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b33145-a82b-4519-ab6a-c7ce0248992f-istiod-ca-cert\") pod \"d5b33145-a82b-4519-ab6a-c7ce0248992f\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " Nov 03 20:26:47.907300 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.907201 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-podinfo\") pod \"d5b33145-a82b-4519-ab6a-c7ce0248992f\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " Nov 03 20:26:47.907398 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.907307 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-workload-certs\") pod \"d5b33145-a82b-4519-ab6a-c7ce0248992f\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " Nov 03 20:26:47.907398 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.907338 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-workload-socket\") pod \"d5b33145-a82b-4519-ab6a-c7ce0248992f\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " Nov 03 20:26:47.907520 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.907496 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-token\" (UniqueName: \"kubernetes.io/projected/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-token\") pod \"d5b33145-a82b-4519-ab6a-c7ce0248992f\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " Nov 03 20:26:47.907630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.907582 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b33145-a82b-4519-ab6a-c7ce0248992f-kube-api-access-jh7qs\" (UniqueName: \"kubernetes.io/projected/d5b33145-a82b-4519-ab6a-c7ce0248992f-kube-api-access-jh7qs\") pod \"d5b33145-a82b-4519-ab6a-c7ce0248992f\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " Nov 03 20:26:47.907630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.907624 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-data\") pod \"d5b33145-a82b-4519-ab6a-c7ce0248992f\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " Nov 03 20:26:47.907779 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.907652 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-envoy\") pod \"d5b33145-a82b-4519-ab6a-c7ce0248992f\" (UID: \"d5b33145-a82b-4519-ab6a-c7ce0248992f\") " Nov 03 20:26:47.913886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.909748 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-workload-certs" pod "d5b33145-a82b-4519-ab6a-c7ce0248992f" (UID: "d5b33145-a82b-4519-ab6a-c7ce0248992f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:47.913886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.910009 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-credential-socket" pod "d5b33145-a82b-4519-ab6a-c7ce0248992f" (UID: "d5b33145-a82b-4519-ab6a-c7ce0248992f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:47.913886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.910309 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b33145-a82b-4519-ab6a-c7ce0248992f-istiod-ca-cert" pod "d5b33145-a82b-4519-ab6a-c7ce0248992f" (UID: "d5b33145-a82b-4519-ab6a-c7ce0248992f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:47.913886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.910920 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-workload-socket" pod "d5b33145-a82b-4519-ab6a-c7ce0248992f" (UID: "d5b33145-a82b-4519-ab6a-c7ce0248992f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:47.913886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.911483 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-data" pod "d5b33145-a82b-4519-ab6a-c7ce0248992f" (UID: "d5b33145-a82b-4519-ab6a-c7ce0248992f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:47.914495 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.914465 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-token" pod "d5b33145-a82b-4519-ab6a-c7ce0248992f" (UID: "d5b33145-a82b-4519-ab6a-c7ce0248992f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:47.917858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.917823 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-podinfo" pod "d5b33145-a82b-4519-ab6a-c7ce0248992f" (UID: "d5b33145-a82b-4519-ab6a-c7ce0248992f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:47.919876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.919837 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b33145-a82b-4519-ab6a-c7ce0248992f-kube-api-access-jh7qs" pod "d5b33145-a82b-4519-ab6a-c7ce0248992f" (UID: "d5b33145-a82b-4519-ab6a-c7ce0248992f"). InnerVolumeSpecName "kube-api-access-jh7qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:47.922753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.922715 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-envoy" pod "d5b33145-a82b-4519-ab6a-c7ce0248992f" (UID: "d5b33145-a82b-4519-ab6a-c7ce0248992f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:47.988933 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:26:47.988896 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd1beb8_21b5_49f9_a228_e2ddf280fc48.slice/crio-d5b9818dceecaae15aa48e8500f00172cc82a15895c1fd6bad90759f57bafa0b WatchSource:0}: Error finding container d5b9818dceecaae15aa48e8500f00172cc82a15895c1fd6bad90759f57bafa0b: Status 404 returned error can't find the container with id d5b9818dceecaae15aa48e8500f00172cc82a15895c1fd6bad90759f57bafa0b Nov 03 20:26:47.989836 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.989724 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7"] Nov 03 20:26:47.992298 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:26:47.992253 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad12f36_4946_4027_a489_7a7f4b6b205b.slice/crio-5b48dd59add6c4d2352bfee63b01bd57ea1ffd5764d9d66e6b6a3eded912b408 WatchSource:0}: Error finding container 5b48dd59add6c4d2352bfee63b01bd57ea1ffd5764d9d66e6b6a3eded912b408: Status 404 returned error can't find the container with id 5b48dd59add6c4d2352bfee63b01bd57ea1ffd5764d9d66e6b6a3eded912b408 Nov 03 20:26:47.997048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:47.997017 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn"] Nov 03 20:26:48.009161 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.009131 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:48.009245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.009166 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b33145-a82b-4519-ab6a-c7ce0248992f-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:48.009245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.009183 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:48.009245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.009196 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:48.009245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.009210 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:48.009245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.009223 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:48.009245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.009237 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh7qs\" (UniqueName: \"kubernetes.io/projected/d5b33145-a82b-4519-ab6a-c7ce0248992f-kube-api-access-jh7qs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:48.009450 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.009252 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:48.009450 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.009266 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b33145-a82b-4519-ab6a-c7ce0248992f-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:48.023082 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.023051 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4"] Nov 03 20:26:48.187476 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.187443 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267cf330-341c-40a3-9b8e-695d9273786a" path="/var/lib/kubelet/pods/267cf330-341c-40a3-9b8e-695d9273786a/volumes" Nov 03 20:26:48.188094 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.188068 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bfeea28-db58-4740-b089-9d8eb0dfece7" path="/var/lib/kubelet/pods/5bfeea28-db58-4740-b089-9d8eb0dfece7/volumes" Nov 03 20:26:48.188592 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.188573 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4" path="/var/lib/kubelet/pods/9619ccf6-8bbb-4c1d-bf4d-675d674ad3d4/volumes" Nov 03 20:26:48.817197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.817117 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" Nov 03 20:26:48.817197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.817126 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn" event={"ID":"d5b33145-a82b-4519-ab6a-c7ce0248992f","Type":"ContainerDied","Data":"0fbdcc6abd5c7cb8a1505766b09e2cbb67de0dad9822c0c3f9a7f76661294ee5"} Nov 03 20:26:48.817197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.817196 5111 scope.go:122] "RemoveContainer" containerID="ad67ec99016939ed0b81e2b594857c37cd515de92a6bfdd357f477b7662cbc9a" Nov 03 20:26:48.818987 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.818956 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" event={"ID":"5dd1beb8-21b5-49f9-a228-e2ddf280fc48","Type":"ContainerStarted","Data":"194b2c902a87e92d98d865c7423c5e31cb6df9c7fb764e513fc867a2cb9fe51f"} Nov 03 20:26:48.819135 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.819019 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" event={"ID":"5dd1beb8-21b5-49f9-a228-e2ddf280fc48","Type":"ContainerStarted","Data":"d5b9818dceecaae15aa48e8500f00172cc82a15895c1fd6bad90759f57bafa0b"} Nov 03 20:26:48.821358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.821325 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" event={"ID":"fad12f36-4946-4027-a489-7a7f4b6b205b","Type":"ContainerStarted","Data":"f53d8d533fa6ead5451ac4a7e6a853f223a1893ba2c0fb6652dc7e138f009ce9"} Nov 03 20:26:48.821486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.821366 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" event={"ID":"fad12f36-4946-4027-a489-7a7f4b6b205b","Type":"ContainerStarted","Data":"5b48dd59add6c4d2352bfee63b01bd57ea1ffd5764d9d66e6b6a3eded912b408"} Nov 03 20:26:48.840001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.839947 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" podStartSLOduration=1.839931014 podStartE2EDuration="1.839931014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:48.838491332 +0000 UTC m=+2838.800869770" watchObservedRunningTime="2026-11-03 20:26:48.839931014 +0000 UTC m=+2838.802309453" Nov 03 20:26:48.863125 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.863091 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn"] Nov 03 20:26:48.867852 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.867812 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-wrjgn"] Nov 03 20:26:48.882423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.882376 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" podStartSLOduration=1.882363426 podStartE2EDuration="1.882363426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:48.878949109 +0000 UTC m=+2838.841327548" watchObservedRunningTime="2026-11-03 20:26:48.882363426 +0000 UTC m=+2838.844741864" Nov 03 20:26:48.990711 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:48.990676 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" Nov 03 20:26:49.024424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.024385 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d2febc-f78a-4494-b202-18f8fdfe80cb-kube-api-access-6d4gd\" (UniqueName: \"kubernetes.io/projected/f5d2febc-f78a-4494-b202-18f8fdfe80cb-kube-api-access-6d4gd\") pod \"f5d2febc-f78a-4494-b202-18f8fdfe80cb\" (UID: \"f5d2febc-f78a-4494-b202-18f8fdfe80cb\") " Nov 03 20:26:49.024661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.024473 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d2febc-f78a-4494-b202-18f8fdfe80cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d2febc-f78a-4494-b202-18f8fdfe80cb-maas-api-openshift-ca\") pod \"f5d2febc-f78a-4494-b202-18f8fdfe80cb\" (UID: \"f5d2febc-f78a-4494-b202-18f8fdfe80cb\") " Nov 03 20:26:49.024745 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.024700 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5d2febc-f78a-4494-b202-18f8fdfe80cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d2febc-f78a-4494-b202-18f8fdfe80cb-maas-api-xks-ca\") pod \"f5d2febc-f78a-4494-b202-18f8fdfe80cb\" (UID: \"f5d2febc-f78a-4494-b202-18f8fdfe80cb\") " Nov 03 20:26:49.024846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.024816 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d2febc-f78a-4494-b202-18f8fdfe80cb-maas-api-openshift-ca" pod "f5d2febc-f78a-4494-b202-18f8fdfe80cb" (UID: "f5d2febc-f78a-4494-b202-18f8fdfe80cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:49.025391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.025364 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d2febc-f78a-4494-b202-18f8fdfe80cb-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:49.027462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.027428 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5d2febc-f78a-4494-b202-18f8fdfe80cb-maas-api-xks-ca" pod "f5d2febc-f78a-4494-b202-18f8fdfe80cb" (UID: "f5d2febc-f78a-4494-b202-18f8fdfe80cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:49.027633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.027523 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d2febc-f78a-4494-b202-18f8fdfe80cb-kube-api-access-6d4gd" pod "f5d2febc-f78a-4494-b202-18f8fdfe80cb" (UID: "f5d2febc-f78a-4494-b202-18f8fdfe80cb"). InnerVolumeSpecName "kube-api-access-6d4gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:49.126641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.126587 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d2febc-f78a-4494-b202-18f8fdfe80cb-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:49.126641 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.126641 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d4gd\" (UniqueName: \"kubernetes.io/projected/f5d2febc-f78a-4494-b202-18f8fdfe80cb-kube-api-access-6d4gd\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:49.674235 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.674211 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:26:49.740833 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.740792 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-workload-socket\") pod \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " Nov 03 20:26:49.741038 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.740892 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-token\" (UniqueName: \"kubernetes.io/projected/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-token\") pod \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " Nov 03 20:26:49.741038 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.740951 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-workload-certs\") pod \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " Nov 03 20:26:49.741038 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.741028 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-envoy\") pod \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " Nov 03 20:26:49.741227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.741064 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-podinfo\") pod \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " Nov 03 20:26:49.741227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.741094 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-data\") pod \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " Nov 03 20:26:49.741227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.741129 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-workload-socket" pod "53ba0307-0ce6-43c8-8599-2fb0cd354f61" (UID: "53ba0307-0ce6-43c8-8599-2fb0cd354f61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:49.741227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.741144 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53ba0307-0ce6-43c8-8599-2fb0cd354f61-kube-api-access-n9vmd\" (UniqueName: \"kubernetes.io/projected/53ba0307-0ce6-43c8-8599-2fb0cd354f61-kube-api-access-n9vmd\") pod \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " Nov 03 20:26:49.741452 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.741267 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istiod-ca-cert\") pod \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " Nov 03 20:26:49.741452 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.741297 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-workload-certs" pod "53ba0307-0ce6-43c8-8599-2fb0cd354f61" (UID: "53ba0307-0ce6-43c8-8599-2fb0cd354f61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:49.741452 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.741380 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-credential-socket\") pod \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\" (UID: \"53ba0307-0ce6-43c8-8599-2fb0cd354f61\") " Nov 03 20:26:49.741677 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.741524 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-data" pod "53ba0307-0ce6-43c8-8599-2fb0cd354f61" (UID: "53ba0307-0ce6-43c8-8599-2fb0cd354f61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:49.741846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.741807 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-credential-socket" pod "53ba0307-0ce6-43c8-8599-2fb0cd354f61" (UID: "53ba0307-0ce6-43c8-8599-2fb0cd354f61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:49.741938 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.741846 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istiod-ca-cert" pod "53ba0307-0ce6-43c8-8599-2fb0cd354f61" (UID: "53ba0307-0ce6-43c8-8599-2fb0cd354f61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:49.742230 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.742206 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:49.742330 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.742234 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:49.742330 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.742250 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:49.742330 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.742265 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:49.742330 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.742278 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:49.744826 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.744585 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-envoy" pod "53ba0307-0ce6-43c8-8599-2fb0cd354f61" (UID: "53ba0307-0ce6-43c8-8599-2fb0cd354f61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:49.744944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.744827 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-podinfo" pod "53ba0307-0ce6-43c8-8599-2fb0cd354f61" (UID: "53ba0307-0ce6-43c8-8599-2fb0cd354f61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:49.745296 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.745269 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-token" pod "53ba0307-0ce6-43c8-8599-2fb0cd354f61" (UID: "53ba0307-0ce6-43c8-8599-2fb0cd354f61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:49.745404 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.745360 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ba0307-0ce6-43c8-8599-2fb0cd354f61-kube-api-access-n9vmd" pod "53ba0307-0ce6-43c8-8599-2fb0cd354f61" (UID: "53ba0307-0ce6-43c8-8599-2fb0cd354f61"). InnerVolumeSpecName "kube-api-access-n9vmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:49.799416 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.799376 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l"] Nov 03 20:26:49.812447 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.812420 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446"] Nov 03 20:26:49.833463 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.833423 5111 generic.go:356] "Generic (PLEG): container finished" podID="5dd1beb8-21b5-49f9-a228-e2ddf280fc48" containerID="194b2c902a87e92d98d865c7423c5e31cb6df9c7fb764e513fc867a2cb9fe51f" exitCode=0 Nov 03 20:26:49.833975 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.833570 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" event={"ID":"5dd1beb8-21b5-49f9-a228-e2ddf280fc48","Type":"ContainerDied","Data":"194b2c902a87e92d98d865c7423c5e31cb6df9c7fb764e513fc867a2cb9fe51f"} Nov 03 20:26:49.835453 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.835432 5111 generic.go:356] "Generic (PLEG): container finished" podID="fad12f36-4946-4027-a489-7a7f4b6b205b" containerID="f53d8d533fa6ead5451ac4a7e6a853f223a1893ba2c0fb6652dc7e138f009ce9" exitCode=0 Nov 03 20:26:49.835596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.835509 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" event={"ID":"fad12f36-4946-4027-a489-7a7f4b6b205b","Type":"ContainerDied","Data":"f53d8d533fa6ead5451ac4a7e6a853f223a1893ba2c0fb6652dc7e138f009ce9"} Nov 03 20:26:49.837263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.837240 5111 generic.go:356] "Generic (PLEG): container finished" podID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" containerID="bde78d8e8312c48a3f78f696d06ee9bf6e7f13ab37f1f5dc1b5824ed2a6e13e0" exitCode=0 Nov 03 20:26:49.837399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.837266 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" event={"ID":"53ba0307-0ce6-43c8-8599-2fb0cd354f61","Type":"ContainerDied","Data":"bde78d8e8312c48a3f78f696d06ee9bf6e7f13ab37f1f5dc1b5824ed2a6e13e0"} Nov 03 20:26:49.837399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.837288 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" Nov 03 20:26:49.837399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.837313 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc" event={"ID":"53ba0307-0ce6-43c8-8599-2fb0cd354f61","Type":"ContainerDied","Data":"371889d8983bba7cf5267237e7c569a5a1b6222ebddfe7e349fe1c8cffc2ccd7"} Nov 03 20:26:49.837399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.837336 5111 scope.go:122] "RemoveContainer" containerID="bde78d8e8312c48a3f78f696d06ee9bf6e7f13ab37f1f5dc1b5824ed2a6e13e0" Nov 03 20:26:49.838858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.838832 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" Nov 03 20:26:49.838965 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.838860 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr" event={"ID":"f5d2febc-f78a-4494-b202-18f8fdfe80cb","Type":"ContainerDied","Data":"1d608794c9f3a751a2f4808f9678d1f277769c4a066b18c6ce6f7f80786580ca"} Nov 03 20:26:49.838965 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.838890 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d608794c9f3a751a2f4808f9678d1f277769c4a066b18c6ce6f7f80786580ca" Nov 03 20:26:49.850730 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.850701 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9vmd\" (UniqueName: \"kubernetes.io/projected/53ba0307-0ce6-43c8-8599-2fb0cd354f61-kube-api-access-n9vmd\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:49.850864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.850735 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:49.850864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.850751 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:49.850864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.850765 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53ba0307-0ce6-43c8-8599-2fb0cd354f61-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:49.858673 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.854155 5111 scope.go:122] "RemoveContainer" containerID="530c601459beaf49d2879234606910984cf2364773161ab50d0305ce6635ecfa" Nov 03 20:26:49.866524 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.866504 5111 scope.go:122] "RemoveContainer" containerID="bde78d8e8312c48a3f78f696d06ee9bf6e7f13ab37f1f5dc1b5824ed2a6e13e0" Nov 03 20:26:49.866840 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:49.866820 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde78d8e8312c48a3f78f696d06ee9bf6e7f13ab37f1f5dc1b5824ed2a6e13e0\": container with ID starting with bde78d8e8312c48a3f78f696d06ee9bf6e7f13ab37f1f5dc1b5824ed2a6e13e0 not found: ID does not exist" containerID="bde78d8e8312c48a3f78f696d06ee9bf6e7f13ab37f1f5dc1b5824ed2a6e13e0" Nov 03 20:26:49.866899 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.866851 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde78d8e8312c48a3f78f696d06ee9bf6e7f13ab37f1f5dc1b5824ed2a6e13e0"} err="failed to get container status \"bde78d8e8312c48a3f78f696d06ee9bf6e7f13ab37f1f5dc1b5824ed2a6e13e0\": rpc error: code = NotFound desc = could not find container \"bde78d8e8312c48a3f78f696d06ee9bf6e7f13ab37f1f5dc1b5824ed2a6e13e0\": container with ID starting with bde78d8e8312c48a3f78f696d06ee9bf6e7f13ab37f1f5dc1b5824ed2a6e13e0 not found: ID does not exist" Nov 03 20:26:49.866899 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.866869 5111 scope.go:122] "RemoveContainer" containerID="530c601459beaf49d2879234606910984cf2364773161ab50d0305ce6635ecfa" Nov 03 20:26:49.867083 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:49.867058 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"530c601459beaf49d2879234606910984cf2364773161ab50d0305ce6635ecfa\": container with ID starting with 530c601459beaf49d2879234606910984cf2364773161ab50d0305ce6635ecfa not found: ID does not exist" containerID="530c601459beaf49d2879234606910984cf2364773161ab50d0305ce6635ecfa" Nov 03 20:26:49.867130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.867090 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"530c601459beaf49d2879234606910984cf2364773161ab50d0305ce6635ecfa"} err="failed to get container status \"530c601459beaf49d2879234606910984cf2364773161ab50d0305ce6635ecfa\": rpc error: code = NotFound desc = could not find container \"530c601459beaf49d2879234606910984cf2364773161ab50d0305ce6635ecfa\": container with ID starting with 530c601459beaf49d2879234606910984cf2364773161ab50d0305ce6635ecfa not found: ID does not exist" Nov 03 20:26:49.912548 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.912450 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc"] Nov 03 20:26:49.916518 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:49.916492 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-944e83-openshift-default-857fc7f84f-pb6gc"] Nov 03 20:26:50.183978 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.183898 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" path="/var/lib/kubelet/pods/53ba0307-0ce6-43c8-8599-2fb0cd354f61/volumes" Nov 03 20:26:50.184633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.184603 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b33145-a82b-4519-ab6a-c7ce0248992f" path="/var/lib/kubelet/pods/d5b33145-a82b-4519-ab6a-c7ce0248992f/volumes" Nov 03 20:26:50.359556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.359518 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:26:50.460157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.460035 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-workload-socket\") pod \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " Nov 03 20:26:50.460157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.460086 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-envoy\") pod \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " Nov 03 20:26:50.460157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.460144 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd161e75-9c5f-4e1c-8a76-5a0171162c42-kube-api-access-nqtj7\" (UniqueName: \"kubernetes.io/projected/dd161e75-9c5f-4e1c-8a76-5a0171162c42-kube-api-access-nqtj7\") pod \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " Nov 03 20:26:50.460469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.460234 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-credential-socket\") pod \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " Nov 03 20:26:50.460469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.460293 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istiod-ca-cert\") pod \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " Nov 03 20:26:50.460469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.460343 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-podinfo\") pod \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " Nov 03 20:26:50.460469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.460375 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-workload-certs\") pod \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " Nov 03 20:26:50.460469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.460465 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-token\" (UniqueName: \"kubernetes.io/projected/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-token\") pod \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " Nov 03 20:26:50.460766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.460518 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-data\") pod \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\" (UID: \"dd161e75-9c5f-4e1c-8a76-5a0171162c42\") " Nov 03 20:26:50.460766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.460632 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-credential-socket" pod "dd161e75-9c5f-4e1c-8a76-5a0171162c42" (UID: "dd161e75-9c5f-4e1c-8a76-5a0171162c42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:50.461763 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.461719 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-workload-certs" pod "dd161e75-9c5f-4e1c-8a76-5a0171162c42" (UID: "dd161e75-9c5f-4e1c-8a76-5a0171162c42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:50.461960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.461910 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-data" pod "dd161e75-9c5f-4e1c-8a76-5a0171162c42" (UID: "dd161e75-9c5f-4e1c-8a76-5a0171162c42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:50.462062 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.461949 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istiod-ca-cert" pod "dd161e75-9c5f-4e1c-8a76-5a0171162c42" (UID: "dd161e75-9c5f-4e1c-8a76-5a0171162c42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:50.462125 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.462049 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-workload-socket" pod "dd161e75-9c5f-4e1c-8a76-5a0171162c42" (UID: "dd161e75-9c5f-4e1c-8a76-5a0171162c42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:50.463601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.463573 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:50.463715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.463610 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:50.463715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.463628 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:50.463715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.463641 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:50.463715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.463660 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:50.468417 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.467043 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-podinfo" pod "dd161e75-9c5f-4e1c-8a76-5a0171162c42" (UID: "dd161e75-9c5f-4e1c-8a76-5a0171162c42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:50.470602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.469683 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd161e75-9c5f-4e1c-8a76-5a0171162c42-kube-api-access-nqtj7" pod "dd161e75-9c5f-4e1c-8a76-5a0171162c42" (UID: "dd161e75-9c5f-4e1c-8a76-5a0171162c42"). InnerVolumeSpecName "kube-api-access-nqtj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:50.480303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.478854 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-token" pod "dd161e75-9c5f-4e1c-8a76-5a0171162c42" (UID: "dd161e75-9c5f-4e1c-8a76-5a0171162c42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:50.481403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.481379 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-envoy" pod "dd161e75-9c5f-4e1c-8a76-5a0171162c42" (UID: "dd161e75-9c5f-4e1c-8a76-5a0171162c42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:50.564732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.564691 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:50.564732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.564725 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:50.564732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.564734 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd161e75-9c5f-4e1c-8a76-5a0171162c42-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:50.564999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.564744 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqtj7\" (UniqueName: \"kubernetes.io/projected/dd161e75-9c5f-4e1c-8a76-5a0171162c42-kube-api-access-nqtj7\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:50.849917 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.849824 5111 generic.go:356] "Generic (PLEG): container finished" podID="dd161e75-9c5f-4e1c-8a76-5a0171162c42" containerID="be379da620d03d13c5d4c536bd4776b855f69a63874df9b83a6ce0c0066b5a82" exitCode=0 Nov 03 20:26:50.849917 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.849907 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" Nov 03 20:26:50.850379 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.849905 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" event={"ID":"dd161e75-9c5f-4e1c-8a76-5a0171162c42","Type":"ContainerDied","Data":"be379da620d03d13c5d4c536bd4776b855f69a63874df9b83a6ce0c0066b5a82"} Nov 03 20:26:50.850379 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.850071 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss" event={"ID":"dd161e75-9c5f-4e1c-8a76-5a0171162c42","Type":"ContainerDied","Data":"e63f30249eb4b1d0e72fda445a0379f40da5231480a5a6721dd5e52da118d424"} Nov 03 20:26:50.850379 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.850103 5111 scope.go:122] "RemoveContainer" containerID="be379da620d03d13c5d4c536bd4776b855f69a63874df9b83a6ce0c0066b5a82" Nov 03 20:26:50.865726 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.865700 5111 scope.go:122] "RemoveContainer" containerID="be379da620d03d13c5d4c536bd4776b855f69a63874df9b83a6ce0c0066b5a82" Nov 03 20:26:50.866064 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:50.866030 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be379da620d03d13c5d4c536bd4776b855f69a63874df9b83a6ce0c0066b5a82\": container with ID starting with be379da620d03d13c5d4c536bd4776b855f69a63874df9b83a6ce0c0066b5a82 not found: ID does not exist" containerID="be379da620d03d13c5d4c536bd4776b855f69a63874df9b83a6ce0c0066b5a82" Nov 03 20:26:50.866188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.866075 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be379da620d03d13c5d4c536bd4776b855f69a63874df9b83a6ce0c0066b5a82"} err="failed to get container status \"be379da620d03d13c5d4c536bd4776b855f69a63874df9b83a6ce0c0066b5a82\": rpc error: code = NotFound desc = could not find container \"be379da620d03d13c5d4c536bd4776b855f69a63874df9b83a6ce0c0066b5a82\": container with ID starting with be379da620d03d13c5d4c536bd4776b855f69a63874df9b83a6ce0c0066b5a82 not found: ID does not exist" Nov 03 20:26:50.871446 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.871406 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" containerID="cri-o://156598ce7679cd5481d4713a9a82a72eb4fde02a1e5f9cf6f546972c8af72fc4" gracePeriod=45 Nov 03 20:26:50.883647 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.883613 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss"] Nov 03 20:26:50.890324 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:50.890294 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-bngss"] Nov 03 20:26:51.038364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.038326 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": dial tcp 10.132.0.101:8000: connect: connection refused" Nov 03 20:26:51.052161 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.052129 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" Nov 03 20:26:51.055695 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.055672 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" Nov 03 20:26:51.189753 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.189725 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-kube-api-access-2tc8h\" (UniqueName: \"kubernetes.io/projected/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-kube-api-access-2tc8h\") pod \"5dd1beb8-21b5-49f9-a228-e2ddf280fc48\" (UID: \"5dd1beb8-21b5-49f9-a228-e2ddf280fc48\") " Nov 03 20:26:51.190052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.190033 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fad12f36-4946-4027-a489-7a7f4b6b205b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad12f36-4946-4027-a489-7a7f4b6b205b-maas-api-xks-ca\") pod \"fad12f36-4946-4027-a489-7a7f4b6b205b\" (UID: \"fad12f36-4946-4027-a489-7a7f4b6b205b\") " Nov 03 20:26:51.190183 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.190166 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad12f36-4946-4027-a489-7a7f4b6b205b-kube-api-access-mctqv\" (UniqueName: \"kubernetes.io/projected/fad12f36-4946-4027-a489-7a7f4b6b205b-kube-api-access-mctqv\") pod \"fad12f36-4946-4027-a489-7a7f4b6b205b\" (UID: \"fad12f36-4946-4027-a489-7a7f4b6b205b\") " Nov 03 20:26:51.190251 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.190199 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad12f36-4946-4027-a489-7a7f4b6b205b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad12f36-4946-4027-a489-7a7f4b6b205b-maas-api-openshift-ca\") pod \"fad12f36-4946-4027-a489-7a7f4b6b205b\" (UID: \"fad12f36-4946-4027-a489-7a7f4b6b205b\") " Nov 03 20:26:51.190251 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.190233 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-maas-api-xks-ca\") pod \"5dd1beb8-21b5-49f9-a228-e2ddf280fc48\" (UID: \"5dd1beb8-21b5-49f9-a228-e2ddf280fc48\") " Nov 03 20:26:51.190383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.190274 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-maas-api-openshift-ca\") pod \"5dd1beb8-21b5-49f9-a228-e2ddf280fc48\" (UID: \"5dd1beb8-21b5-49f9-a228-e2ddf280fc48\") " Nov 03 20:26:51.191007 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.190979 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-maas-api-openshift-ca" pod "5dd1beb8-21b5-49f9-a228-e2ddf280fc48" (UID: "5dd1beb8-21b5-49f9-a228-e2ddf280fc48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:51.191647 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.191620 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad12f36-4946-4027-a489-7a7f4b6b205b-maas-api-openshift-ca" pod "fad12f36-4946-4027-a489-7a7f4b6b205b" (UID: "fad12f36-4946-4027-a489-7a7f4b6b205b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:51.193978 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.193949 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad12f36-4946-4027-a489-7a7f4b6b205b-kube-api-access-mctqv" pod "fad12f36-4946-4027-a489-7a7f4b6b205b" (UID: "fad12f36-4946-4027-a489-7a7f4b6b205b"). InnerVolumeSpecName "kube-api-access-mctqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:51.194355 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.194315 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-kube-api-access-2tc8h" pod "5dd1beb8-21b5-49f9-a228-e2ddf280fc48" (UID: "5dd1beb8-21b5-49f9-a228-e2ddf280fc48"). InnerVolumeSpecName "kube-api-access-2tc8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:51.194870 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.194850 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad12f36-4946-4027-a489-7a7f4b6b205b-maas-api-xks-ca" pod "fad12f36-4946-4027-a489-7a7f4b6b205b" (UID: "fad12f36-4946-4027-a489-7a7f4b6b205b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:51.196440 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.196417 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-maas-api-xks-ca" pod "5dd1beb8-21b5-49f9-a228-e2ddf280fc48" (UID: "5dd1beb8-21b5-49f9-a228-e2ddf280fc48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:51.221759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.221729 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:26:51.292020 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.291985 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b53a56a-77d0-4fbe-b767-82563feb20a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/8b53a56a-77d0-4fbe-b767-82563feb20a0-tls-certs\") pod \"8b53a56a-77d0-4fbe-b767-82563feb20a0\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " Nov 03 20:26:51.292276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.292169 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b53a56a-77d0-4fbe-b767-82563feb20a0-kube-api-access-5vdd4\" (UniqueName: \"kubernetes.io/projected/8b53a56a-77d0-4fbe-b767-82563feb20a0-kube-api-access-5vdd4\") pod \"8b53a56a-77d0-4fbe-b767-82563feb20a0\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " Nov 03 20:26:51.292353 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.292235 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-dshm\") pod \"8b53a56a-77d0-4fbe-b767-82563feb20a0\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " Nov 03 20:26:51.292478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.292454 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-model-cache\") pod \"8b53a56a-77d0-4fbe-b767-82563feb20a0\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " Nov 03 20:26:51.292648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.292517 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-tmp-dir\") pod \"8b53a56a-77d0-4fbe-b767-82563feb20a0\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " Nov 03 20:26:51.292648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.292599 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-home\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-home\") pod \"8b53a56a-77d0-4fbe-b767-82563feb20a0\" (UID: \"8b53a56a-77d0-4fbe-b767-82563feb20a0\") " Nov 03 20:26:51.293142 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.292890 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-model-cache" pod "8b53a56a-77d0-4fbe-b767-82563feb20a0" (UID: "8b53a56a-77d0-4fbe-b767-82563feb20a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:51.293142 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.293114 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-tmp-dir" pod "8b53a56a-77d0-4fbe-b767-82563feb20a0" (UID: "8b53a56a-77d0-4fbe-b767-82563feb20a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:51.293142 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.293129 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-home" pod "8b53a56a-77d0-4fbe-b767-82563feb20a0" (UID: "8b53a56a-77d0-4fbe-b767-82563feb20a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:51.293467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.293222 5111 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-model-cache\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.293467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.293243 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tc8h\" (UniqueName: \"kubernetes.io/projected/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-kube-api-access-2tc8h\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.293467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.293259 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad12f36-4946-4027-a489-7a7f4b6b205b-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.293467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.293273 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mctqv\" (UniqueName: \"kubernetes.io/projected/fad12f36-4946-4027-a489-7a7f4b6b205b-kube-api-access-mctqv\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.293467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.293286 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad12f36-4946-4027-a489-7a7f4b6b205b-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.293467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.293301 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.293467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.293314 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd1beb8-21b5-49f9-a228-e2ddf280fc48-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.295011 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.294979 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b53a56a-77d0-4fbe-b767-82563feb20a0-tls-certs" pod "8b53a56a-77d0-4fbe-b767-82563feb20a0" (UID: "8b53a56a-77d0-4fbe-b767-82563feb20a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:51.295130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.295039 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b53a56a-77d0-4fbe-b767-82563feb20a0-kube-api-access-5vdd4" pod "8b53a56a-77d0-4fbe-b767-82563feb20a0" (UID: "8b53a56a-77d0-4fbe-b767-82563feb20a0"). InnerVolumeSpecName "kube-api-access-5vdd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:51.295364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.295335 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-dshm" pod "8b53a56a-77d0-4fbe-b767-82563feb20a0" (UID: "8b53a56a-77d0-4fbe-b767-82563feb20a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:51.395153 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.395056 5111 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-home\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.395153 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.395091 5111 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b53a56a-77d0-4fbe-b767-82563feb20a0-tls-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.395153 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.395104 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vdd4\" (UniqueName: \"kubernetes.io/projected/8b53a56a-77d0-4fbe-b767-82563feb20a0-kube-api-access-5vdd4\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.395153 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.395118 5111 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-dshm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.395153 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.395130 5111 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b53a56a-77d0-4fbe-b767-82563feb20a0-tmp-dir\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.859031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.858943 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" Nov 03 20:26:51.859031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.858982 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7" event={"ID":"5dd1beb8-21b5-49f9-a228-e2ddf280fc48","Type":"ContainerDied","Data":"d5b9818dceecaae15aa48e8500f00172cc82a15895c1fd6bad90759f57bafa0b"} Nov 03 20:26:51.859520 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.859043 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5b9818dceecaae15aa48e8500f00172cc82a15895c1fd6bad90759f57bafa0b" Nov 03 20:26:51.860433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.860414 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" Nov 03 20:26:51.860605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.860441 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn" event={"ID":"fad12f36-4946-4027-a489-7a7f4b6b205b","Type":"ContainerDied","Data":"5b48dd59add6c4d2352bfee63b01bd57ea1ffd5764d9d66e6b6a3eded912b408"} Nov 03 20:26:51.860605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.860481 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b48dd59add6c4d2352bfee63b01bd57ea1ffd5764d9d66e6b6a3eded912b408" Nov 03 20:26:51.863050 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.863026 5111 generic.go:356] "Generic (PLEG): container finished" podID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerID="156598ce7679cd5481d4713a9a82a72eb4fde02a1e5f9cf6f546972c8af72fc4" exitCode=0 Nov 03 20:26:51.863158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.863061 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" Nov 03 20:26:51.863158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.863101 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" event={"ID":"8b53a56a-77d0-4fbe-b767-82563feb20a0","Type":"ContainerDied","Data":"156598ce7679cd5481d4713a9a82a72eb4fde02a1e5f9cf6f546972c8af72fc4"} Nov 03 20:26:51.863158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.863144 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz" event={"ID":"8b53a56a-77d0-4fbe-b767-82563feb20a0","Type":"ContainerDied","Data":"2d1140584a063fbb73f815afdff9d8bea53869ed22628e52a70187dedf5d4d2c"} Nov 03 20:26:51.863312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.863166 5111 scope.go:122] "RemoveContainer" containerID="156598ce7679cd5481d4713a9a82a72eb4fde02a1e5f9cf6f546972c8af72fc4" Nov 03 20:26:51.876777 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.876763 5111 scope.go:122] "RemoveContainer" containerID="156598ce7679cd5481d4713a9a82a72eb4fde02a1e5f9cf6f546972c8af72fc4" Nov 03 20:26:51.876992 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:51.876973 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"156598ce7679cd5481d4713a9a82a72eb4fde02a1e5f9cf6f546972c8af72fc4\": container with ID starting with 156598ce7679cd5481d4713a9a82a72eb4fde02a1e5f9cf6f546972c8af72fc4 not found: ID does not exist" containerID="156598ce7679cd5481d4713a9a82a72eb4fde02a1e5f9cf6f546972c8af72fc4" Nov 03 20:26:51.877050 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.876998 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156598ce7679cd5481d4713a9a82a72eb4fde02a1e5f9cf6f546972c8af72fc4"} err="failed to get container status \"156598ce7679cd5481d4713a9a82a72eb4fde02a1e5f9cf6f546972c8af72fc4\": rpc error: code = NotFound desc = could not find container \"156598ce7679cd5481d4713a9a82a72eb4fde02a1e5f9cf6f546972c8af72fc4\": container with ID starting with 156598ce7679cd5481d4713a9a82a72eb4fde02a1e5f9cf6f546972c8af72fc4 not found: ID does not exist" Nov 03 20:26:51.895266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.895238 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz"] Nov 03 20:26:51.900520 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:51.900499 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/distinct-w0-120fe3-kserve-86b546d956-5lxfz"] Nov 03 20:26:52.184120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:52.184083 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" path="/var/lib/kubelet/pods/8b53a56a-77d0-4fbe-b767-82563feb20a0/volumes" Nov 03 20:26:52.184704 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:52.184688 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd161e75-9c5f-4e1c-8a76-5a0171162c42" path="/var/lib/kubelet/pods/dd161e75-9c5f-4e1c-8a76-5a0171162c42/volumes" Nov 03 20:26:53.407428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:53.407386 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6"] Nov 03 20:26:54.044752 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.044715 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz"] Nov 03 20:26:54.045369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045348 5111 cpu_manager.go:414] "containerMap: removing container" podUID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" containerName="istio-proxy" Nov 03 20:26:54.045369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045367 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" containerName="istio-proxy" Nov 03 20:26:54.045579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045413 5111 cpu_manager.go:414] "containerMap: removing container" podUID="d5b33145-a82b-4519-ab6a-c7ce0248992f" containerName="istio-proxy" Nov 03 20:26:54.045579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045422 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b33145-a82b-4519-ab6a-c7ce0248992f" containerName="istio-proxy" Nov 03 20:26:54.045579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045450 5111 cpu_manager.go:414] "containerMap: removing container" podUID="dd161e75-9c5f-4e1c-8a76-5a0171162c42" containerName="istio-proxy" Nov 03 20:26:54.045579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045456 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd161e75-9c5f-4e1c-8a76-5a0171162c42" containerName="istio-proxy" Nov 03 20:26:54.045579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045462 5111 cpu_manager.go:414] "containerMap: removing container" podUID="f5d2febc-f78a-4494-b202-18f8fdfe80cb" containerName="revoke-keys" Nov 03 20:26:54.045579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045467 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d2febc-f78a-4494-b202-18f8fdfe80cb" containerName="revoke-keys" Nov 03 20:26:54.045579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045474 5111 cpu_manager.go:414] "containerMap: removing container" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" Nov 03 20:26:54.045579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045479 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" Nov 03 20:26:54.045579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045499 5111 cpu_manager.go:414] "containerMap: removing container" podUID="5dd1beb8-21b5-49f9-a228-e2ddf280fc48" containerName="revoke-keys" Nov 03 20:26:54.045579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045505 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd1beb8-21b5-49f9-a228-e2ddf280fc48" containerName="revoke-keys" Nov 03 20:26:54.045579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045523 5111 cpu_manager.go:414] "containerMap: removing container" podUID="fad12f36-4946-4027-a489-7a7f4b6b205b" containerName="revoke-keys" Nov 03 20:26:54.045579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045528 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad12f36-4946-4027-a489-7a7f4b6b205b" containerName="revoke-keys" Nov 03 20:26:54.046004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045650 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b33145-a82b-4519-ab6a-c7ce0248992f" containerName="istio-proxy" Nov 03 20:26:54.046004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045661 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd161e75-9c5f-4e1c-8a76-5a0171162c42" containerName="istio-proxy" Nov 03 20:26:54.046004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045667 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d2febc-f78a-4494-b202-18f8fdfe80cb" containerName="revoke-keys" Nov 03 20:26:54.046004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045672 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b53a56a-77d0-4fbe-b767-82563feb20a0" containerName="main" Nov 03 20:26:54.046004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045680 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd1beb8-21b5-49f9-a228-e2ddf280fc48" containerName="revoke-keys" Nov 03 20:26:54.046004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045688 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad12f36-4946-4027-a489-7a7f4b6b205b" containerName="revoke-keys" Nov 03 20:26:54.046004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045695 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" containerName="istio-proxy" Nov 03 20:26:54.046004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.045705 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" containerName="istio-proxy" Nov 03 20:26:54.051269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.050919 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz" Nov 03 20:26:54.053453 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.053246 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q"] Nov 03 20:26:54.054045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.054021 5111 cpu_manager.go:414] "containerMap: removing container" podUID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" containerName="istio-proxy" Nov 03 20:26:54.054152 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.054047 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ba0307-0ce6-43c8-8599-2fb0cd354f61" containerName="istio-proxy" Nov 03 20:26:54.054866 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.054844 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:26:54.055168 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.055151 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:26:54.055388 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.055365 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:26:54.055494 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.055460 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:26:54.058486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.058465 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz"] Nov 03 20:26:54.058662 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.058636 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q" Nov 03 20:26:54.082837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.082804 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q"] Nov 03 20:26:54.145037 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.144989 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-466xl\" (UniqueName: \"kubernetes.io/projected/77848c07-7655-4d10-9a19-76e07bc77e8a-kube-api-access-466xl\") pod \"maas-api-revoke-sub-14290fa9-4z2bz\" (UID: \"77848c07-7655-4d10-9a19-76e07bc77e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz" Nov 03 20:26:54.145203 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.145064 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77848c07-7655-4d10-9a19-76e07bc77e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14290fa9-4z2bz\" (UID: \"77848c07-7655-4d10-9a19-76e07bc77e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz" Nov 03 20:26:54.145203 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.145145 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc60303a-7c35-40e2-a552-4ddf22fa666f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4ca16b1-6wg6q\" (UID: \"bc60303a-7c35-40e2-a552-4ddf22fa666f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q" Nov 03 20:26:54.145386 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.145364 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc60303a-7c35-40e2-a552-4ddf22fa666f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4ca16b1-6wg6q\" (UID: \"bc60303a-7c35-40e2-a552-4ddf22fa666f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q" Nov 03 20:26:54.145434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.145403 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77848c07-7655-4d10-9a19-76e07bc77e8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14290fa9-4z2bz\" (UID: \"77848c07-7655-4d10-9a19-76e07bc77e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz" Nov 03 20:26:54.145594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.145569 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkjs8\" (UniqueName: \"kubernetes.io/projected/bc60303a-7c35-40e2-a552-4ddf22fa666f-kube-api-access-nkjs8\") pod \"maas-api-revoke-sub-c4ca16b1-6wg6q\" (UID: \"bc60303a-7c35-40e2-a552-4ddf22fa666f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q" Nov 03 20:26:54.246616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.246567 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc60303a-7c35-40e2-a552-4ddf22fa666f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4ca16b1-6wg6q\" (UID: \"bc60303a-7c35-40e2-a552-4ddf22fa666f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q" Nov 03 20:26:54.246851 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.246629 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77848c07-7655-4d10-9a19-76e07bc77e8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14290fa9-4z2bz\" (UID: \"77848c07-7655-4d10-9a19-76e07bc77e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz" Nov 03 20:26:54.246851 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.246709 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkjs8\" (UniqueName: \"kubernetes.io/projected/bc60303a-7c35-40e2-a552-4ddf22fa666f-kube-api-access-nkjs8\") pod \"maas-api-revoke-sub-c4ca16b1-6wg6q\" (UID: \"bc60303a-7c35-40e2-a552-4ddf22fa666f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q" Nov 03 20:26:54.246851 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.246750 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-466xl\" (UniqueName: \"kubernetes.io/projected/77848c07-7655-4d10-9a19-76e07bc77e8a-kube-api-access-466xl\") pod \"maas-api-revoke-sub-14290fa9-4z2bz\" (UID: \"77848c07-7655-4d10-9a19-76e07bc77e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz" Nov 03 20:26:54.246851 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.246800 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77848c07-7655-4d10-9a19-76e07bc77e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14290fa9-4z2bz\" (UID: \"77848c07-7655-4d10-9a19-76e07bc77e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz" Nov 03 20:26:54.246851 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.246836 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc60303a-7c35-40e2-a552-4ddf22fa666f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4ca16b1-6wg6q\" (UID: \"bc60303a-7c35-40e2-a552-4ddf22fa666f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q" Nov 03 20:26:54.247467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.247431 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc60303a-7c35-40e2-a552-4ddf22fa666f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4ca16b1-6wg6q\" (UID: \"bc60303a-7c35-40e2-a552-4ddf22fa666f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q" Nov 03 20:26:54.247771 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.247749 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77848c07-7655-4d10-9a19-76e07bc77e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14290fa9-4z2bz\" (UID: \"77848c07-7655-4d10-9a19-76e07bc77e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz" Nov 03 20:26:54.249620 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.249575 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc60303a-7c35-40e2-a552-4ddf22fa666f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4ca16b1-6wg6q\" (UID: \"bc60303a-7c35-40e2-a552-4ddf22fa666f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q" Nov 03 20:26:54.249718 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.249681 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77848c07-7655-4d10-9a19-76e07bc77e8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14290fa9-4z2bz\" (UID: \"77848c07-7655-4d10-9a19-76e07bc77e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz" Nov 03 20:26:54.255104 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.255073 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkjs8\" (UniqueName: \"kubernetes.io/projected/bc60303a-7c35-40e2-a552-4ddf22fa666f-kube-api-access-nkjs8\") pod \"maas-api-revoke-sub-c4ca16b1-6wg6q\" (UID: \"bc60303a-7c35-40e2-a552-4ddf22fa666f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q" Nov 03 20:26:54.255203 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.255138 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-466xl\" (UniqueName: \"kubernetes.io/projected/77848c07-7655-4d10-9a19-76e07bc77e8a-kube-api-access-466xl\") pod \"maas-api-revoke-sub-14290fa9-4z2bz\" (UID: \"77848c07-7655-4d10-9a19-76e07bc77e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz" Nov 03 20:26:54.388445 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.388413 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz" Nov 03 20:26:54.396787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.396765 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q" Nov 03 20:26:54.689609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.689582 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz"] Nov 03 20:26:54.690424 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:26:54.690395 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77848c07_7655_4d10_9a19_76e07bc77e8a.slice/crio-b5477184e13d1ef825d52cc6be1063663913a099614bce1c730dd610ac4a5ad9 WatchSource:0}: Error finding container b5477184e13d1ef825d52cc6be1063663913a099614bce1c730dd610ac4a5ad9: Status 404 returned error can't find the container with id b5477184e13d1ef825d52cc6be1063663913a099614bce1c730dd610ac4a5ad9 Nov 03 20:26:54.696072 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.696048 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q"] Nov 03 20:26:54.696673 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:26:54.696651 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc60303a_7c35_40e2_a552_4ddf22fa666f.slice/crio-f131ff3aaa45aa83ba4313aa73f09e6cfac460eba7a21277e05598b31db29f99 WatchSource:0}: Error finding container f131ff3aaa45aa83ba4313aa73f09e6cfac460eba7a21277e05598b31db29f99: Status 404 returned error can't find the container with id f131ff3aaa45aa83ba4313aa73f09e6cfac460eba7a21277e05598b31db29f99 Nov 03 20:26:54.889094 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.889048 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q" event={"ID":"bc60303a-7c35-40e2-a552-4ddf22fa666f","Type":"ContainerStarted","Data":"f131ff3aaa45aa83ba4313aa73f09e6cfac460eba7a21277e05598b31db29f99"} Nov 03 20:26:54.890472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:54.890443 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz" event={"ID":"77848c07-7655-4d10-9a19-76e07bc77e8a","Type":"ContainerStarted","Data":"b5477184e13d1ef825d52cc6be1063663913a099614bce1c730dd610ac4a5ad9"} Nov 03 20:26:55.119622 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:55.119585 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j"] Nov 03 20:26:55.472323 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:55.472285 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f"] Nov 03 20:26:55.472586 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:55.472556 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" podUID="5478802b-2542-47ee-919d-5a8ad79a5e6a" containerName="istio-proxy" containerID="cri-o://44fe7c39f39f7049fe7af14e26eaf9541523be6e3ca1a9b143d96a7c21035236" gracePeriod=30 Nov 03 20:26:55.900927 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:55.900894 5111 generic.go:356] "Generic (PLEG): container finished" podID="bc60303a-7c35-40e2-a552-4ddf22fa666f" containerID="3586e96c785bf27c5192f9f79ebe5408d255153080836c9bd75a6af59650ca06" exitCode=0 Nov 03 20:26:55.901383 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:55.901031 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q" event={"ID":"bc60303a-7c35-40e2-a552-4ddf22fa666f","Type":"ContainerDied","Data":"3586e96c785bf27c5192f9f79ebe5408d255153080836c9bd75a6af59650ca06"} Nov 03 20:26:55.902984 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:55.902961 5111 generic.go:356] "Generic (PLEG): container finished" podID="77848c07-7655-4d10-9a19-76e07bc77e8a" containerID="2462ba8a187fba6a55ea84ae8feed52eb3cf9da5ed7f34551b9ba0e800b338cc" exitCode=0 Nov 03 20:26:55.903109 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:55.903043 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz" event={"ID":"77848c07-7655-4d10-9a19-76e07bc77e8a","Type":"ContainerDied","Data":"2462ba8a187fba6a55ea84ae8feed52eb3cf9da5ed7f34551b9ba0e800b338cc"} Nov 03 20:26:56.080642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:56.080615 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" Nov 03 20:26:56.103707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:56.103671 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f79a1068-5b22-4a40-8c5a-1565c2593369-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f79a1068-5b22-4a40-8c5a-1565c2593369-plugins-config-volume\") pod \"f79a1068-5b22-4a40-8c5a-1565c2593369\" (UID: \"f79a1068-5b22-4a40-8c5a-1565c2593369\") " Nov 03 20:26:56.103707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:56.103717 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79a1068-5b22-4a40-8c5a-1565c2593369-kube-api-access-zwqq2\" (UniqueName: \"kubernetes.io/projected/f79a1068-5b22-4a40-8c5a-1565c2593369-kube-api-access-zwqq2\") pod \"f79a1068-5b22-4a40-8c5a-1565c2593369\" (UID: \"f79a1068-5b22-4a40-8c5a-1565c2593369\") " Nov 03 20:26:56.104156 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:56.104120 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79a1068-5b22-4a40-8c5a-1565c2593369-plugins-config-volume" pod "f79a1068-5b22-4a40-8c5a-1565c2593369" (UID: "f79a1068-5b22-4a40-8c5a-1565c2593369"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:56.104349 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:56.104328 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f79a1068-5b22-4a40-8c5a-1565c2593369-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:56.106498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:56.106467 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79a1068-5b22-4a40-8c5a-1565c2593369-kube-api-access-zwqq2" pod "f79a1068-5b22-4a40-8c5a-1565c2593369" (UID: "f79a1068-5b22-4a40-8c5a-1565c2593369"). InnerVolumeSpecName "kube-api-access-zwqq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:56.205149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:56.205108 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwqq2\" (UniqueName: \"kubernetes.io/projected/f79a1068-5b22-4a40-8c5a-1565c2593369-kube-api-access-zwqq2\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:56.927726 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:56.927687 5111 generic.go:356] "Generic (PLEG): container finished" podID="f79a1068-5b22-4a40-8c5a-1565c2593369" containerID="39351c34f51037b782b36380bd64c1490c1be7277f0b7a4c4a1642248c5c7e39" exitCode=0 Nov 03 20:26:56.928249 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:56.927748 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" Nov 03 20:26:56.928249 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:56.927773 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" event={"ID":"f79a1068-5b22-4a40-8c5a-1565c2593369","Type":"ContainerDied","Data":"39351c34f51037b782b36380bd64c1490c1be7277f0b7a4c4a1642248c5c7e39"} Nov 03 20:26:56.928249 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:56.927875 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs" event={"ID":"f79a1068-5b22-4a40-8c5a-1565c2593369","Type":"ContainerDied","Data":"597050db3a70b1acaa0adbbdd6e5ff01604b195608054756d077d3752011da39"} Nov 03 20:26:56.928249 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:56.927898 5111 scope.go:122] "RemoveContainer" containerID="39351c34f51037b782b36380bd64c1490c1be7277f0b7a4c4a1642248c5c7e39" Nov 03 20:26:56.941701 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:56.941679 5111 scope.go:122] "RemoveContainer" containerID="39351c34f51037b782b36380bd64c1490c1be7277f0b7a4c4a1642248c5c7e39" Nov 03 20:26:56.942072 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:26:56.942044 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39351c34f51037b782b36380bd64c1490c1be7277f0b7a4c4a1642248c5c7e39\": container with ID starting with 39351c34f51037b782b36380bd64c1490c1be7277f0b7a4c4a1642248c5c7e39 not found: ID does not exist" containerID="39351c34f51037b782b36380bd64c1490c1be7277f0b7a4c4a1642248c5c7e39" Nov 03 20:26:56.942172 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:56.942083 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39351c34f51037b782b36380bd64c1490c1be7277f0b7a4c4a1642248c5c7e39"} err="failed to get container status \"39351c34f51037b782b36380bd64c1490c1be7277f0b7a4c4a1642248c5c7e39\": rpc error: code = NotFound desc = could not find container \"39351c34f51037b782b36380bd64c1490c1be7277f0b7a4c4a1642248c5c7e39\": container with ID starting with 39351c34f51037b782b36380bd64c1490c1be7277f0b7a4c4a1642248c5c7e39 not found: ID does not exist" Nov 03 20:26:56.947048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:56.947017 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs"] Nov 03 20:26:56.954861 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:56.954831 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-27d3ad-596dfb9455-9hlzs"] Nov 03 20:26:57.076784 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.073872 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd"] Nov 03 20:26:57.076784 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.074753 5111 cpu_manager.go:414] "containerMap: removing container" podUID="f79a1068-5b22-4a40-8c5a-1565c2593369" containerName="payload-pre-processing" Nov 03 20:26:57.076784 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.074775 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f79a1068-5b22-4a40-8c5a-1565c2593369" containerName="payload-pre-processing" Nov 03 20:26:57.076784 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.074977 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="f79a1068-5b22-4a40-8c5a-1565c2593369" containerName="payload-pre-processing" Nov 03 20:26:57.081603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.080825 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd" Nov 03 20:26:57.083341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.082376 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd"] Nov 03 20:26:57.123601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.123574 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e0c2bf-68f5-4cbd-a110-c24180dd4525-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a64755c9-sgtfd\" (UID: \"76e0c2bf-68f5-4cbd-a110-c24180dd4525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd" Nov 03 20:26:57.123862 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.123693 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e0c2bf-68f5-4cbd-a110-c24180dd4525-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a64755c9-sgtfd\" (UID: \"76e0c2bf-68f5-4cbd-a110-c24180dd4525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd" Nov 03 20:26:57.123862 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.123781 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkvg7\" (UniqueName: \"kubernetes.io/projected/76e0c2bf-68f5-4cbd-a110-c24180dd4525-kube-api-access-zkvg7\") pod \"maas-api-revoke-sub-a64755c9-sgtfd\" (UID: \"76e0c2bf-68f5-4cbd-a110-c24180dd4525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd" Nov 03 20:26:57.138457 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.138433 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz" Nov 03 20:26:57.143426 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.143405 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q" Nov 03 20:26:57.225103 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.225017 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc60303a-7c35-40e2-a552-4ddf22fa666f-kube-api-access-nkjs8\" (UniqueName: \"kubernetes.io/projected/bc60303a-7c35-40e2-a552-4ddf22fa666f-kube-api-access-nkjs8\") pod \"bc60303a-7c35-40e2-a552-4ddf22fa666f\" (UID: \"bc60303a-7c35-40e2-a552-4ddf22fa666f\") " Nov 03 20:26:57.225290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.225131 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77848c07-7655-4d10-9a19-76e07bc77e8a-kube-api-access-466xl\" (UniqueName: \"kubernetes.io/projected/77848c07-7655-4d10-9a19-76e07bc77e8a-kube-api-access-466xl\") pod \"77848c07-7655-4d10-9a19-76e07bc77e8a\" (UID: \"77848c07-7655-4d10-9a19-76e07bc77e8a\") " Nov 03 20:26:57.225290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.225155 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc60303a-7c35-40e2-a552-4ddf22fa666f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc60303a-7c35-40e2-a552-4ddf22fa666f-maas-api-xks-ca\") pod \"bc60303a-7c35-40e2-a552-4ddf22fa666f\" (UID: \"bc60303a-7c35-40e2-a552-4ddf22fa666f\") " Nov 03 20:26:57.225290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.225182 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc60303a-7c35-40e2-a552-4ddf22fa666f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc60303a-7c35-40e2-a552-4ddf22fa666f-maas-api-openshift-ca\") pod \"bc60303a-7c35-40e2-a552-4ddf22fa666f\" (UID: \"bc60303a-7c35-40e2-a552-4ddf22fa666f\") " Nov 03 20:26:57.225290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.225205 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77848c07-7655-4d10-9a19-76e07bc77e8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77848c07-7655-4d10-9a19-76e07bc77e8a-maas-api-openshift-ca\") pod \"77848c07-7655-4d10-9a19-76e07bc77e8a\" (UID: \"77848c07-7655-4d10-9a19-76e07bc77e8a\") " Nov 03 20:26:57.225567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.225330 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77848c07-7655-4d10-9a19-76e07bc77e8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77848c07-7655-4d10-9a19-76e07bc77e8a-maas-api-xks-ca\") pod \"77848c07-7655-4d10-9a19-76e07bc77e8a\" (UID: \"77848c07-7655-4d10-9a19-76e07bc77e8a\") " Nov 03 20:26:57.225642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.225588 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e0c2bf-68f5-4cbd-a110-c24180dd4525-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a64755c9-sgtfd\" (UID: \"76e0c2bf-68f5-4cbd-a110-c24180dd4525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd" Nov 03 20:26:57.225642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.225607 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc60303a-7c35-40e2-a552-4ddf22fa666f-maas-api-openshift-ca" pod "bc60303a-7c35-40e2-a552-4ddf22fa666f" (UID: "bc60303a-7c35-40e2-a552-4ddf22fa666f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:57.225747 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.225629 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77848c07-7655-4d10-9a19-76e07bc77e8a-maas-api-openshift-ca" pod "77848c07-7655-4d10-9a19-76e07bc77e8a" (UID: "77848c07-7655-4d10-9a19-76e07bc77e8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:57.225972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.225946 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkvg7\" (UniqueName: \"kubernetes.io/projected/76e0c2bf-68f5-4cbd-a110-c24180dd4525-kube-api-access-zkvg7\") pod \"maas-api-revoke-sub-a64755c9-sgtfd\" (UID: \"76e0c2bf-68f5-4cbd-a110-c24180dd4525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd" Nov 03 20:26:57.226234 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.226213 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e0c2bf-68f5-4cbd-a110-c24180dd4525-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a64755c9-sgtfd\" (UID: \"76e0c2bf-68f5-4cbd-a110-c24180dd4525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd" Nov 03 20:26:57.226408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.226384 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc60303a-7c35-40e2-a552-4ddf22fa666f-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:57.226408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.226410 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77848c07-7655-4d10-9a19-76e07bc77e8a-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:57.227105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.227078 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e0c2bf-68f5-4cbd-a110-c24180dd4525-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a64755c9-sgtfd\" (UID: \"76e0c2bf-68f5-4cbd-a110-c24180dd4525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd" Nov 03 20:26:57.227788 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.227749 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77848c07-7655-4d10-9a19-76e07bc77e8a-maas-api-xks-ca" pod "77848c07-7655-4d10-9a19-76e07bc77e8a" (UID: "77848c07-7655-4d10-9a19-76e07bc77e8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:57.227788 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.227760 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc60303a-7c35-40e2-a552-4ddf22fa666f-kube-api-access-nkjs8" pod "bc60303a-7c35-40e2-a552-4ddf22fa666f" (UID: "bc60303a-7c35-40e2-a552-4ddf22fa666f"). InnerVolumeSpecName "kube-api-access-nkjs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:57.227972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.227951 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc60303a-7c35-40e2-a552-4ddf22fa666f-maas-api-xks-ca" pod "bc60303a-7c35-40e2-a552-4ddf22fa666f" (UID: "bc60303a-7c35-40e2-a552-4ddf22fa666f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:57.228030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.227986 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77848c07-7655-4d10-9a19-76e07bc77e8a-kube-api-access-466xl" pod "77848c07-7655-4d10-9a19-76e07bc77e8a" (UID: "77848c07-7655-4d10-9a19-76e07bc77e8a"). InnerVolumeSpecName "kube-api-access-466xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:57.228088 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.228053 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e0c2bf-68f5-4cbd-a110-c24180dd4525-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a64755c9-sgtfd\" (UID: \"76e0c2bf-68f5-4cbd-a110-c24180dd4525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd" Nov 03 20:26:57.233857 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.233837 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkvg7\" (UniqueName: \"kubernetes.io/projected/76e0c2bf-68f5-4cbd-a110-c24180dd4525-kube-api-access-zkvg7\") pod \"maas-api-revoke-sub-a64755c9-sgtfd\" (UID: \"76e0c2bf-68f5-4cbd-a110-c24180dd4525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd" Nov 03 20:26:57.327403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.327361 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77848c07-7655-4d10-9a19-76e07bc77e8a-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:57.327403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.327398 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkjs8\" (UniqueName: \"kubernetes.io/projected/bc60303a-7c35-40e2-a552-4ddf22fa666f-kube-api-access-nkjs8\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:57.327403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.327409 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-466xl\" (UniqueName: \"kubernetes.io/projected/77848c07-7655-4d10-9a19-76e07bc77e8a-kube-api-access-466xl\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:57.327688 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.327418 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc60303a-7c35-40e2-a552-4ddf22fa666f-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:57.440205 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.440175 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd" Nov 03 20:26:57.724678 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.724642 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd"] Nov 03 20:26:57.725275 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:26:57.725242 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e0c2bf_68f5_4cbd_a110_c24180dd4525.slice/crio-90eaaad2a6a2fec311e0f019c2bfceffea0cc7b9e5dff89d0a475e56c16f134d WatchSource:0}: Error finding container 90eaaad2a6a2fec311e0f019c2bfceffea0cc7b9e5dff89d0a475e56c16f134d: Status 404 returned error can't find the container with id 90eaaad2a6a2fec311e0f019c2bfceffea0cc7b9e5dff89d0a475e56c16f134d Nov 03 20:26:57.937989 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.937961 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz" Nov 03 20:26:57.938414 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.937989 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz" event={"ID":"77848c07-7655-4d10-9a19-76e07bc77e8a","Type":"ContainerDied","Data":"b5477184e13d1ef825d52cc6be1063663913a099614bce1c730dd610ac4a5ad9"} Nov 03 20:26:57.938414 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.938040 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5477184e13d1ef825d52cc6be1063663913a099614bce1c730dd610ac4a5ad9" Nov 03 20:26:57.940895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.940868 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd" event={"ID":"76e0c2bf-68f5-4cbd-a110-c24180dd4525","Type":"ContainerStarted","Data":"90eaaad2a6a2fec311e0f019c2bfceffea0cc7b9e5dff89d0a475e56c16f134d"} Nov 03 20:26:57.942444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.942423 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q" event={"ID":"bc60303a-7c35-40e2-a552-4ddf22fa666f","Type":"ContainerDied","Data":"f131ff3aaa45aa83ba4313aa73f09e6cfac460eba7a21277e05598b31db29f99"} Nov 03 20:26:57.942542 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.942447 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q" Nov 03 20:26:57.942542 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:57.942459 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f131ff3aaa45aa83ba4313aa73f09e6cfac460eba7a21277e05598b31db29f99" Nov 03 20:26:58.187190 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:58.187159 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f79a1068-5b22-4a40-8c5a-1565c2593369" path="/var/lib/kubelet/pods/f79a1068-5b22-4a40-8c5a-1565c2593369/volumes" Nov 03 20:26:58.407862 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:58.407744 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" podUID="67f55725-b707-46f2-9cf5-8932b47df3b8" containerName="payload-processing" containerID="cri-o://8306cc4bbe5d8c8e75626e264e1ab35c1dac9aa40f6cface6df09adedd14978a" gracePeriod=25 Nov 03 20:26:58.951368 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:58.951288 5111 generic.go:356] "Generic (PLEG): container finished" podID="76e0c2bf-68f5-4cbd-a110-c24180dd4525" containerID="c483b88f4ae76b281dc5583ea64d1df1897fb8b2c0f00527101874fdc3711bc9" exitCode=0 Nov 03 20:26:58.951805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:26:58.951374 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd" event={"ID":"76e0c2bf-68f5-4cbd-a110-c24180dd4525","Type":"ContainerDied","Data":"c483b88f4ae76b281dc5583ea64d1df1897fb8b2c0f00527101874fdc3711bc9"} Nov 03 20:27:00.120893 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.120687 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" podUID="8d4c64f2-4b08-4847-ade9-8035b3f7853c" containerName="payload-pre-processing" containerID="cri-o://a73901a8b96750605632061a49b742f4d6e607b3571d12dd648a954dbb923474" gracePeriod=25 Nov 03 20:27:00.242293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.242263 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd" Nov 03 20:27:00.265515 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.265481 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76e0c2bf-68f5-4cbd-a110-c24180dd4525-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e0c2bf-68f5-4cbd-a110-c24180dd4525-maas-api-xks-ca\") pod \"76e0c2bf-68f5-4cbd-a110-c24180dd4525\" (UID: \"76e0c2bf-68f5-4cbd-a110-c24180dd4525\") " Nov 03 20:27:00.265694 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.265644 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76e0c2bf-68f5-4cbd-a110-c24180dd4525-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e0c2bf-68f5-4cbd-a110-c24180dd4525-maas-api-openshift-ca\") pod \"76e0c2bf-68f5-4cbd-a110-c24180dd4525\" (UID: \"76e0c2bf-68f5-4cbd-a110-c24180dd4525\") " Nov 03 20:27:00.265755 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.265733 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e0c2bf-68f5-4cbd-a110-c24180dd4525-kube-api-access-zkvg7\" (UniqueName: \"kubernetes.io/projected/76e0c2bf-68f5-4cbd-a110-c24180dd4525-kube-api-access-zkvg7\") pod \"76e0c2bf-68f5-4cbd-a110-c24180dd4525\" (UID: \"76e0c2bf-68f5-4cbd-a110-c24180dd4525\") " Nov 03 20:27:00.266053 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.266027 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e0c2bf-68f5-4cbd-a110-c24180dd4525-maas-api-openshift-ca" pod "76e0c2bf-68f5-4cbd-a110-c24180dd4525" (UID: "76e0c2bf-68f5-4cbd-a110-c24180dd4525"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:27:00.266738 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.266704 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e0c2bf-68f5-4cbd-a110-c24180dd4525-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:00.268349 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.268323 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76e0c2bf-68f5-4cbd-a110-c24180dd4525-maas-api-xks-ca" pod "76e0c2bf-68f5-4cbd-a110-c24180dd4525" (UID: "76e0c2bf-68f5-4cbd-a110-c24180dd4525"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:27:00.268466 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.268375 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e0c2bf-68f5-4cbd-a110-c24180dd4525-kube-api-access-zkvg7" pod "76e0c2bf-68f5-4cbd-a110-c24180dd4525" (UID: "76e0c2bf-68f5-4cbd-a110-c24180dd4525"). InnerVolumeSpecName "kube-api-access-zkvg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:00.277295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.277265 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq"] Nov 03 20:27:00.292575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.290459 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-120fe3-rtzsq"] Nov 03 20:27:00.367859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.367808 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkvg7\" (UniqueName: \"kubernetes.io/projected/76e0c2bf-68f5-4cbd-a110-c24180dd4525-kube-api-access-zkvg7\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:00.367859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.367855 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e0c2bf-68f5-4cbd-a110-c24180dd4525-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:00.864415 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.864377 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:27:00.879020 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.878567 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-podinfo\") pod \"5478802b-2542-47ee-919d-5a8ad79a5e6a\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " Nov 03 20:27:00.879214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.879119 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-token\" (UniqueName: \"kubernetes.io/projected/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-token\") pod \"5478802b-2542-47ee-919d-5a8ad79a5e6a\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " Nov 03 20:27:00.879863 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.879544 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-envoy\") pod \"5478802b-2542-47ee-919d-5a8ad79a5e6a\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " Nov 03 20:27:00.879863 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.879614 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-workload-certs\") pod \"5478802b-2542-47ee-919d-5a8ad79a5e6a\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " Nov 03 20:27:00.879863 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.879671 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-credential-socket\") pod \"5478802b-2542-47ee-919d-5a8ad79a5e6a\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " Nov 03 20:27:00.879863 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.879759 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-workload-socket\") pod \"5478802b-2542-47ee-919d-5a8ad79a5e6a\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " Nov 03 20:27:00.879863 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.879789 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-data\") pod \"5478802b-2542-47ee-919d-5a8ad79a5e6a\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " Nov 03 20:27:00.879863 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.879820 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5478802b-2542-47ee-919d-5a8ad79a5e6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5478802b-2542-47ee-919d-5a8ad79a5e6a-istiod-ca-cert\") pod \"5478802b-2542-47ee-919d-5a8ad79a5e6a\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " Nov 03 20:27:00.879863 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.879867 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5478802b-2542-47ee-919d-5a8ad79a5e6a-kube-api-access-mmsqf\" (UniqueName: \"kubernetes.io/projected/5478802b-2542-47ee-919d-5a8ad79a5e6a-kube-api-access-mmsqf\") pod \"5478802b-2542-47ee-919d-5a8ad79a5e6a\" (UID: \"5478802b-2542-47ee-919d-5a8ad79a5e6a\") " Nov 03 20:27:00.881038 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.880956 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-workload-certs" pod "5478802b-2542-47ee-919d-5a8ad79a5e6a" (UID: "5478802b-2542-47ee-919d-5a8ad79a5e6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:00.881292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.881219 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-workload-socket" pod "5478802b-2542-47ee-919d-5a8ad79a5e6a" (UID: "5478802b-2542-47ee-919d-5a8ad79a5e6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:00.881292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.881258 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-data" pod "5478802b-2542-47ee-919d-5a8ad79a5e6a" (UID: "5478802b-2542-47ee-919d-5a8ad79a5e6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:00.881626 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.881504 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-credential-socket" pod "5478802b-2542-47ee-919d-5a8ad79a5e6a" (UID: "5478802b-2542-47ee-919d-5a8ad79a5e6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:00.881890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.881855 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5478802b-2542-47ee-919d-5a8ad79a5e6a-istiod-ca-cert" pod "5478802b-2542-47ee-919d-5a8ad79a5e6a" (UID: "5478802b-2542-47ee-919d-5a8ad79a5e6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:27:00.891616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.891080 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-podinfo" pod "5478802b-2542-47ee-919d-5a8ad79a5e6a" (UID: "5478802b-2542-47ee-919d-5a8ad79a5e6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:27:00.891616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.891577 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-token" pod "5478802b-2542-47ee-919d-5a8ad79a5e6a" (UID: "5478802b-2542-47ee-919d-5a8ad79a5e6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:00.892944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.892906 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-envoy" pod "5478802b-2542-47ee-919d-5a8ad79a5e6a" (UID: "5478802b-2542-47ee-919d-5a8ad79a5e6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:00.905722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.905679 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5478802b-2542-47ee-919d-5a8ad79a5e6a-kube-api-access-mmsqf" pod "5478802b-2542-47ee-919d-5a8ad79a5e6a" (UID: "5478802b-2542-47ee-919d-5a8ad79a5e6a"). InnerVolumeSpecName "kube-api-access-mmsqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.973290 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd" event={"ID":"76e0c2bf-68f5-4cbd-a110-c24180dd4525","Type":"ContainerDied","Data":"90eaaad2a6a2fec311e0f019c2bfceffea0cc7b9e5dff89d0a475e56c16f134d"} Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.973353 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90eaaad2a6a2fec311e0f019c2bfceffea0cc7b9e5dff89d0a475e56c16f134d" Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.973427 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd" Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.979554 5111 generic.go:356] "Generic (PLEG): container finished" podID="5478802b-2542-47ee-919d-5a8ad79a5e6a" containerID="44fe7c39f39f7049fe7af14e26eaf9541523be6e3ca1a9b143d96a7c21035236" exitCode=0 Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.979639 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.979640 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" event={"ID":"5478802b-2542-47ee-919d-5a8ad79a5e6a","Type":"ContainerDied","Data":"44fe7c39f39f7049fe7af14e26eaf9541523be6e3ca1a9b143d96a7c21035236"} Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.979727 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f" event={"ID":"5478802b-2542-47ee-919d-5a8ad79a5e6a","Type":"ContainerDied","Data":"8db3373dd0b90cf3383d55f563683c8c450d0220d9cc3131c93f141e4d4f554e"} Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.979757 5111 scope.go:122] "RemoveContainer" containerID="44fe7c39f39f7049fe7af14e26eaf9541523be6e3ca1a9b143d96a7c21035236" Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.981111 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.981128 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.981137 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.981147 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.981156 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.981165 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.981177 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5478802b-2542-47ee-919d-5a8ad79a5e6a-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.981186 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5478802b-2542-47ee-919d-5a8ad79a5e6a-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:00.982360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.981195 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmsqf\" (UniqueName: \"kubernetes.io/projected/5478802b-2542-47ee-919d-5a8ad79a5e6a-kube-api-access-mmsqf\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:00.998996 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:00.998971 5111 scope.go:122] "RemoveContainer" containerID="041b235c029964f198d7c3e2af1e79a8f57e3c3b45d4528370b72a2cac25709b" Nov 03 20:27:01.010724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:01.010700 5111 scope.go:122] "RemoveContainer" containerID="44fe7c39f39f7049fe7af14e26eaf9541523be6e3ca1a9b143d96a7c21035236" Nov 03 20:27:01.011076 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:01.011047 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44fe7c39f39f7049fe7af14e26eaf9541523be6e3ca1a9b143d96a7c21035236\": container with ID starting with 44fe7c39f39f7049fe7af14e26eaf9541523be6e3ca1a9b143d96a7c21035236 not found: ID does not exist" containerID="44fe7c39f39f7049fe7af14e26eaf9541523be6e3ca1a9b143d96a7c21035236" Nov 03 20:27:01.011167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:01.011083 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44fe7c39f39f7049fe7af14e26eaf9541523be6e3ca1a9b143d96a7c21035236"} err="failed to get container status \"44fe7c39f39f7049fe7af14e26eaf9541523be6e3ca1a9b143d96a7c21035236\": rpc error: code = NotFound desc = could not find container \"44fe7c39f39f7049fe7af14e26eaf9541523be6e3ca1a9b143d96a7c21035236\": container with ID starting with 44fe7c39f39f7049fe7af14e26eaf9541523be6e3ca1a9b143d96a7c21035236 not found: ID does not exist" Nov 03 20:27:01.011167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:01.011101 5111 scope.go:122] "RemoveContainer" containerID="041b235c029964f198d7c3e2af1e79a8f57e3c3b45d4528370b72a2cac25709b" Nov 03 20:27:01.011418 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:01.011400 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"041b235c029964f198d7c3e2af1e79a8f57e3c3b45d4528370b72a2cac25709b\": container with ID starting with 041b235c029964f198d7c3e2af1e79a8f57e3c3b45d4528370b72a2cac25709b not found: ID does not exist" containerID="041b235c029964f198d7c3e2af1e79a8f57e3c3b45d4528370b72a2cac25709b" Nov 03 20:27:01.011474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:01.011431 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"041b235c029964f198d7c3e2af1e79a8f57e3c3b45d4528370b72a2cac25709b"} err="failed to get container status \"041b235c029964f198d7c3e2af1e79a8f57e3c3b45d4528370b72a2cac25709b\": rpc error: code = NotFound desc = could not find container \"041b235c029964f198d7c3e2af1e79a8f57e3c3b45d4528370b72a2cac25709b\": container with ID starting with 041b235c029964f198d7c3e2af1e79a8f57e3c3b45d4528370b72a2cac25709b not found: ID does not exist" Nov 03 20:27:01.019886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:01.019857 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f"] Nov 03 20:27:01.027777 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:01.027745 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-d5b7f"] Nov 03 20:27:02.187778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:02.187748 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38304073-d13f-4d9c-ac90-81ba88cb910b" path="/var/lib/kubelet/pods/38304073-d13f-4d9c-ac90-81ba88cb910b/volumes" Nov 03 20:27:02.188288 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:02.188274 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5478802b-2542-47ee-919d-5a8ad79a5e6a" path="/var/lib/kubelet/pods/5478802b-2542-47ee-919d-5a8ad79a5e6a/volumes" Nov 03 20:27:03.047890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.047836 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" containerID="cri-o://87a653f7cc8f20e0e426f7ebc307d0f19138d0887301168396d1850f951f60fe" gracePeriod=45 Nov 03 20:27:03.329818 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.329788 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:27:03.418858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.418821 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-home\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-home\") pod \"afbc0e79-dd6e-4c42-9330-ff7988817e80\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " Nov 03 20:27:03.419054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.418884 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afbc0e79-dd6e-4c42-9330-ff7988817e80-kube-api-access-sqxpg\" (UniqueName: \"kubernetes.io/projected/afbc0e79-dd6e-4c42-9330-ff7988817e80-kube-api-access-sqxpg\") pod \"afbc0e79-dd6e-4c42-9330-ff7988817e80\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " Nov 03 20:27:03.419054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.418924 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afbc0e79-dd6e-4c42-9330-ff7988817e80-tls-certs\" (UniqueName: \"kubernetes.io/secret/afbc0e79-dd6e-4c42-9330-ff7988817e80-tls-certs\") pod \"afbc0e79-dd6e-4c42-9330-ff7988817e80\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " Nov 03 20:27:03.419054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.419015 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-dshm\") pod \"afbc0e79-dd6e-4c42-9330-ff7988817e80\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " Nov 03 20:27:03.419054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.419048 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-model-cache\") pod \"afbc0e79-dd6e-4c42-9330-ff7988817e80\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " Nov 03 20:27:03.419302 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.419200 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-home" pod "afbc0e79-dd6e-4c42-9330-ff7988817e80" (UID: "afbc0e79-dd6e-4c42-9330-ff7988817e80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:03.419302 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.419256 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-tmp-dir\") pod \"afbc0e79-dd6e-4c42-9330-ff7988817e80\" (UID: \"afbc0e79-dd6e-4c42-9330-ff7988817e80\") " Nov 03 20:27:03.419728 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.419694 5111 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-home\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:03.419977 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.419957 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-tmp-dir" pod "afbc0e79-dd6e-4c42-9330-ff7988817e80" (UID: "afbc0e79-dd6e-4c42-9330-ff7988817e80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:03.420160 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.420139 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-model-cache" pod "afbc0e79-dd6e-4c42-9330-ff7988817e80" (UID: "afbc0e79-dd6e-4c42-9330-ff7988817e80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:03.422185 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.422162 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-dshm" pod "afbc0e79-dd6e-4c42-9330-ff7988817e80" (UID: "afbc0e79-dd6e-4c42-9330-ff7988817e80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:03.422305 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.422252 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afbc0e79-dd6e-4c42-9330-ff7988817e80-tls-certs" pod "afbc0e79-dd6e-4c42-9330-ff7988817e80" (UID: "afbc0e79-dd6e-4c42-9330-ff7988817e80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:27:03.430493 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.430468 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afbc0e79-dd6e-4c42-9330-ff7988817e80-kube-api-access-sqxpg" pod "afbc0e79-dd6e-4c42-9330-ff7988817e80" (UID: "afbc0e79-dd6e-4c42-9330-ff7988817e80"). InnerVolumeSpecName "kube-api-access-sqxpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:03.521309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.521273 5111 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-dshm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:03.521309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.521304 5111 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-model-cache\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:03.521309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.521314 5111 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbc0e79-dd6e-4c42-9330-ff7988817e80-tmp-dir\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:03.521591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.521323 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqxpg\" (UniqueName: \"kubernetes.io/projected/afbc0e79-dd6e-4c42-9330-ff7988817e80-kube-api-access-sqxpg\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:03.521591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:03.521335 5111 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afbc0e79-dd6e-4c42-9330-ff7988817e80-tls-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:04.008124 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:04.008091 5111 generic.go:356] "Generic (PLEG): container finished" podID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerID="87a653f7cc8f20e0e426f7ebc307d0f19138d0887301168396d1850f951f60fe" exitCode=0 Nov 03 20:27:04.008330 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:04.008148 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" Nov 03 20:27:04.008330 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:04.008175 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" event={"ID":"afbc0e79-dd6e-4c42-9330-ff7988817e80","Type":"ContainerDied","Data":"87a653f7cc8f20e0e426f7ebc307d0f19138d0887301168396d1850f951f60fe"} Nov 03 20:27:04.008330 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:04.008225 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4" event={"ID":"afbc0e79-dd6e-4c42-9330-ff7988817e80","Type":"ContainerDied","Data":"166fdb35a5b3c1f225cae705fdc21d98fbf822d9f570fc77bc76944d093c0e50"} Nov 03 20:27:04.008330 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:04.008242 5111 scope.go:122] "RemoveContainer" containerID="87a653f7cc8f20e0e426f7ebc307d0f19138d0887301168396d1850f951f60fe" Nov 03 20:27:04.020819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:04.020799 5111 scope.go:122] "RemoveContainer" containerID="87a653f7cc8f20e0e426f7ebc307d0f19138d0887301168396d1850f951f60fe" Nov 03 20:27:04.021132 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:04.021112 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a653f7cc8f20e0e426f7ebc307d0f19138d0887301168396d1850f951f60fe\": container with ID starting with 87a653f7cc8f20e0e426f7ebc307d0f19138d0887301168396d1850f951f60fe not found: ID does not exist" containerID="87a653f7cc8f20e0e426f7ebc307d0f19138d0887301168396d1850f951f60fe" Nov 03 20:27:04.021193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:04.021144 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a653f7cc8f20e0e426f7ebc307d0f19138d0887301168396d1850f951f60fe"} err="failed to get container status \"87a653f7cc8f20e0e426f7ebc307d0f19138d0887301168396d1850f951f60fe\": rpc error: code = NotFound desc = could not find container \"87a653f7cc8f20e0e426f7ebc307d0f19138d0887301168396d1850f951f60fe\": container with ID starting with 87a653f7cc8f20e0e426f7ebc307d0f19138d0887301168396d1850f951f60fe not found: ID does not exist" Nov 03 20:27:04.033240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:04.033207 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4"] Nov 03 20:27:04.037035 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:04.037014 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/simulator-w5-ab1e46-kserve-7988b48557-lt2w4"] Nov 03 20:27:04.183680 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:04.183644 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" path="/var/lib/kubelet/pods/afbc0e79-dd6e-4c42-9330-ff7988817e80/volumes" Nov 03 20:27:04.822739 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:04.822677 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" containerID="cri-o://9795e1c7a47dd18c7b53e7fe23b8e952d5a0bc8c0c98a2a580e4e4264edb52cd" gracePeriod=45 Nov 03 20:27:04.836291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:04.836221 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" containerID="cri-o://165652cafb9f442391bee353f07035570477f7d2c9ac2f7c114ad616eedfd14b" gracePeriod=45 Nov 03 20:27:05.022262 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.022225 5111 generic.go:356] "Generic (PLEG): container finished" podID="7959c647-ef80-40d4-8262-335b9e72934b" containerID="165652cafb9f442391bee353f07035570477f7d2c9ac2f7c114ad616eedfd14b" exitCode=0 Nov 03 20:27:05.022457 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.022372 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" event={"ID":"7959c647-ef80-40d4-8262-335b9e72934b","Type":"ContainerDied","Data":"165652cafb9f442391bee353f07035570477f7d2c9ac2f7c114ad616eedfd14b"} Nov 03 20:27:05.024467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.024436 5111 generic.go:356] "Generic (PLEG): container finished" podID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerID="9795e1c7a47dd18c7b53e7fe23b8e952d5a0bc8c0c98a2a580e4e4264edb52cd" exitCode=0 Nov 03 20:27:05.024645 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.024557 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" event={"ID":"ec0ff9ad-04a1-4709-960f-78e78bbaca58","Type":"ContainerDied","Data":"9795e1c7a47dd18c7b53e7fe23b8e952d5a0bc8c0c98a2a580e4e4264edb52cd"} Nov 03 20:27:05.067097 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:05.066858 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:27:05.089809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.089721 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" podUID="8d4c64f2-4b08-4847-ade9-8035b3f7853c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:27:05.089809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.089795 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" podUID="67f55725-b707-46f2-9cf5-8932b47df3b8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:27:05.136747 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.136719 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:27:05.170901 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.170871 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:27:05.245658 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.245616 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-dshm\") pod \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " Nov 03 20:27:05.245824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.245695 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-model-cache\") pod \"7959c647-ef80-40d4-8262-335b9e72934b\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " Nov 03 20:27:05.245824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.245739 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0ff9ad-04a1-4709-960f-78e78bbaca58-kube-api-access-tjpww\" (UniqueName: \"kubernetes.io/projected/ec0ff9ad-04a1-4709-960f-78e78bbaca58-kube-api-access-tjpww\") pod \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " Nov 03 20:27:05.245824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.245801 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec0ff9ad-04a1-4709-960f-78e78bbaca58-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0ff9ad-04a1-4709-960f-78e78bbaca58-tls-certs\") pod \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " Nov 03 20:27:05.246005 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.245834 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7959c647-ef80-40d4-8262-335b9e72934b-kube-api-access-9vrqm\" (UniqueName: \"kubernetes.io/projected/7959c647-ef80-40d4-8262-335b9e72934b-kube-api-access-9vrqm\") pod \"7959c647-ef80-40d4-8262-335b9e72934b\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " Nov 03 20:27:05.246005 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.245977 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-model-cache\") pod \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " Nov 03 20:27:05.246120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.245999 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-model-cache" pod "7959c647-ef80-40d4-8262-335b9e72934b" (UID: "7959c647-ef80-40d4-8262-335b9e72934b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:05.246120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.246085 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-home\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-home\") pod \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " Nov 03 20:27:05.246239 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.246191 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7959c647-ef80-40d4-8262-335b9e72934b-tls-certs\" (UniqueName: \"kubernetes.io/secret/7959c647-ef80-40d4-8262-335b9e72934b-tls-certs\") pod \"7959c647-ef80-40d4-8262-335b9e72934b\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " Nov 03 20:27:05.246288 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.246252 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-dshm\") pod \"7959c647-ef80-40d4-8262-335b9e72934b\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " Nov 03 20:27:05.246340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.246304 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-home\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-home\") pod \"7959c647-ef80-40d4-8262-335b9e72934b\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " Nov 03 20:27:05.246340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.246331 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-tmp-dir\") pod \"7959c647-ef80-40d4-8262-335b9e72934b\" (UID: \"7959c647-ef80-40d4-8262-335b9e72934b\") " Nov 03 20:27:05.246453 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.246352 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-home" pod "ec0ff9ad-04a1-4709-960f-78e78bbaca58" (UID: "ec0ff9ad-04a1-4709-960f-78e78bbaca58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:05.246453 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.246386 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-tmp-dir\") pod \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\" (UID: \"ec0ff9ad-04a1-4709-960f-78e78bbaca58\") " Nov 03 20:27:05.248034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.247018 5111 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-model-cache\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:05.248034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.247429 5111 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-home\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:05.248034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.247559 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-tmp-dir" pod "7959c647-ef80-40d4-8262-335b9e72934b" (UID: "7959c647-ef80-40d4-8262-335b9e72934b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:05.248034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.247781 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-home" pod "7959c647-ef80-40d4-8262-335b9e72934b" (UID: "7959c647-ef80-40d4-8262-335b9e72934b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:05.248401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.248347 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-model-cache" pod "ec0ff9ad-04a1-4709-960f-78e78bbaca58" (UID: "ec0ff9ad-04a1-4709-960f-78e78bbaca58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:05.248560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.248513 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-tmp-dir" pod "ec0ff9ad-04a1-4709-960f-78e78bbaca58" (UID: "ec0ff9ad-04a1-4709-960f-78e78bbaca58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:05.248785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.248764 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0ff9ad-04a1-4709-960f-78e78bbaca58-kube-api-access-tjpww" pod "ec0ff9ad-04a1-4709-960f-78e78bbaca58" (UID: "ec0ff9ad-04a1-4709-960f-78e78bbaca58"). InnerVolumeSpecName "kube-api-access-tjpww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:05.248861 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.248760 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec0ff9ad-04a1-4709-960f-78e78bbaca58-tls-certs" pod "ec0ff9ad-04a1-4709-960f-78e78bbaca58" (UID: "ec0ff9ad-04a1-4709-960f-78e78bbaca58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:27:05.248998 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.248964 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-dshm" pod "ec0ff9ad-04a1-4709-960f-78e78bbaca58" (UID: "ec0ff9ad-04a1-4709-960f-78e78bbaca58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:05.249124 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.249005 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7959c647-ef80-40d4-8262-335b9e72934b-kube-api-access-9vrqm" pod "7959c647-ef80-40d4-8262-335b9e72934b" (UID: "7959c647-ef80-40d4-8262-335b9e72934b"). InnerVolumeSpecName "kube-api-access-9vrqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:05.249292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.249275 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-dshm" pod "7959c647-ef80-40d4-8262-335b9e72934b" (UID: "7959c647-ef80-40d4-8262-335b9e72934b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:05.250213 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.250188 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7959c647-ef80-40d4-8262-335b9e72934b-tls-certs" pod "7959c647-ef80-40d4-8262-335b9e72934b" (UID: "7959c647-ef80-40d4-8262-335b9e72934b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:27:05.348716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.348638 5111 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7959c647-ef80-40d4-8262-335b9e72934b-tls-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:05.348716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.348670 5111 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-dshm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:05.348716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.348679 5111 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-home\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:05.348716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.348688 5111 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7959c647-ef80-40d4-8262-335b9e72934b-tmp-dir\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:05.348716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.348696 5111 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-tmp-dir\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:05.348716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.348704 5111 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-dshm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:05.348716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.348715 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjpww\" (UniqueName: \"kubernetes.io/projected/ec0ff9ad-04a1-4709-960f-78e78bbaca58-kube-api-access-tjpww\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:05.348716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.348723 5111 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0ff9ad-04a1-4709-960f-78e78bbaca58-tls-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:05.349195 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.348732 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vrqm\" (UniqueName: \"kubernetes.io/projected/7959c647-ef80-40d4-8262-335b9e72934b-kube-api-access-9vrqm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:05.349195 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:05.348742 5111 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec0ff9ad-04a1-4709-960f-78e78bbaca58-model-cache\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:06.035319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:06.035278 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" event={"ID":"7959c647-ef80-40d4-8262-335b9e72934b","Type":"ContainerDied","Data":"b5fd76c8245305e9146acca9bc512ca361e68624763f109d41ac8dde9ce7d73a"} Nov 03 20:27:06.035319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:06.035314 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446" Nov 03 20:27:06.035927 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:06.035341 5111 scope.go:122] "RemoveContainer" containerID="165652cafb9f442391bee353f07035570477f7d2c9ac2f7c114ad616eedfd14b" Nov 03 20:27:06.037077 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:06.036984 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" Nov 03 20:27:06.037214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:06.037113 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l" event={"ID":"ec0ff9ad-04a1-4709-960f-78e78bbaca58","Type":"ContainerDied","Data":"5222a82f347a966efce5d28fc270f6eb63f3306ead1924c3f53c3a36e8eb43d3"} Nov 03 20:27:06.052364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:06.052332 5111 scope.go:122] "RemoveContainer" containerID="9795e1c7a47dd18c7b53e7fe23b8e952d5a0bc8c0c98a2a580e4e4264edb52cd" Nov 03 20:27:06.066759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:06.066736 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446"] Nov 03 20:27:06.073517 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:06.073494 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/simulator-w1-b42aea-kserve-7f68f546b7-s7446"] Nov 03 20:27:06.092940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:06.087159 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb"] Nov 03 20:27:06.092940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:06.092868 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l"] Nov 03 20:27:06.095322 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:06.095298 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/premium-w1-b42aea-kserve-5d994f8d94-tqg7l"] Nov 03 20:27:06.184567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:06.184504 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7959c647-ef80-40d4-8262-335b9e72934b" path="/var/lib/kubelet/pods/7959c647-ef80-40d4-8262-335b9e72934b/volumes" Nov 03 20:27:06.185095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:06.185076 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" path="/var/lib/kubelet/pods/ec0ff9ad-04a1-4709-960f-78e78bbaca58/volumes" Nov 03 20:27:08.068465 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:08.068423 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j"] Nov 03 20:27:08.076254 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:08.076225 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ab1e46-bc82j"] Nov 03 20:27:08.166097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:08.166059 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z"] Nov 03 20:27:08.166625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:08.166567 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" podUID="96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" containerName="istio-proxy" containerID="cri-o://5af8b136c308d412ed77327969f64975d358c108ce260bed9134c16312cb742f" gracePeriod=30 Nov 03 20:27:08.190780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:08.190739 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62ed539-4a84-4b9e-bd92-d93c67e928b4" path="/var/lib/kubelet/pods/e62ed539-4a84-4b9e-bd92-d93c67e928b4/volumes" Nov 03 20:27:09.919174 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:09.919139 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr"] Nov 03 20:27:09.922195 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:09.922170 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b42aea-96qkr"] Nov 03 20:27:10.183140 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:10.183058 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5d2febc-f78a-4494-b202-18f8fdfe80cb" path="/var/lib/kubelet/pods/f5d2febc-f78a-4494-b202-18f8fdfe80cb/volumes" Nov 03 20:27:11.087507 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:11.087467 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" podUID="6f695bca-daf7-4b45-baab-e4b6a5c5adbc" containerName="payload-pre-processing" containerID="cri-o://3a61961c9210b987d9c50a2a3f7fe7304f25a45f10ac5626e320ed274fe22257" gracePeriod=25 Nov 03 20:27:11.362069 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:11.361983 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" Nov 03 20:27:11.445514 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:11.445477 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f695bca-daf7-4b45-baab-e4b6a5c5adbc-kube-api-access-f8dwb\" (UniqueName: \"kubernetes.io/projected/6f695bca-daf7-4b45-baab-e4b6a5c5adbc-kube-api-access-f8dwb\") pod \"6f695bca-daf7-4b45-baab-e4b6a5c5adbc\" (UID: \"6f695bca-daf7-4b45-baab-e4b6a5c5adbc\") " Nov 03 20:27:11.445514 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:11.445520 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f695bca-daf7-4b45-baab-e4b6a5c5adbc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f695bca-daf7-4b45-baab-e4b6a5c5adbc-plugins-config-volume\") pod \"6f695bca-daf7-4b45-baab-e4b6a5c5adbc\" (UID: \"6f695bca-daf7-4b45-baab-e4b6a5c5adbc\") " Nov 03 20:27:11.446007 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:11.445978 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f695bca-daf7-4b45-baab-e4b6a5c5adbc-plugins-config-volume" pod "6f695bca-daf7-4b45-baab-e4b6a5c5adbc" (UID: "6f695bca-daf7-4b45-baab-e4b6a5c5adbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:27:11.446163 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:11.446068 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f695bca-daf7-4b45-baab-e4b6a5c5adbc-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:11.448062 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:11.448032 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f695bca-daf7-4b45-baab-e4b6a5c5adbc-kube-api-access-f8dwb" pod "6f695bca-daf7-4b45-baab-e4b6a5c5adbc" (UID: "6f695bca-daf7-4b45-baab-e4b6a5c5adbc"). InnerVolumeSpecName "kube-api-access-f8dwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:11.547230 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:11.547192 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8dwb\" (UniqueName: \"kubernetes.io/projected/6f695bca-daf7-4b45-baab-e4b6a5c5adbc-kube-api-access-f8dwb\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:12.096404 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:12.096369 5111 generic.go:356] "Generic (PLEG): container finished" podID="6f695bca-daf7-4b45-baab-e4b6a5c5adbc" containerID="3a61961c9210b987d9c50a2a3f7fe7304f25a45f10ac5626e320ed274fe22257" exitCode=0 Nov 03 20:27:12.096883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:12.096416 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" Nov 03 20:27:12.096883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:12.096447 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" event={"ID":"6f695bca-daf7-4b45-baab-e4b6a5c5adbc","Type":"ContainerDied","Data":"3a61961c9210b987d9c50a2a3f7fe7304f25a45f10ac5626e320ed274fe22257"} Nov 03 20:27:12.096883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:12.096508 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb" event={"ID":"6f695bca-daf7-4b45-baab-e4b6a5c5adbc","Type":"ContainerDied","Data":"394bcc5a6d2f3c4098c4a0186dc8d092c92f4b2232ba1c8ff5f471dbf5037a0d"} Nov 03 20:27:12.096883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:12.096551 5111 scope.go:122] "RemoveContainer" containerID="3a61961c9210b987d9c50a2a3f7fe7304f25a45f10ac5626e320ed274fe22257" Nov 03 20:27:12.113472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:12.113436 5111 scope.go:122] "RemoveContainer" containerID="3a61961c9210b987d9c50a2a3f7fe7304f25a45f10ac5626e320ed274fe22257" Nov 03 20:27:12.113806 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:12.113778 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a61961c9210b987d9c50a2a3f7fe7304f25a45f10ac5626e320ed274fe22257\": container with ID starting with 3a61961c9210b987d9c50a2a3f7fe7304f25a45f10ac5626e320ed274fe22257 not found: ID does not exist" containerID="3a61961c9210b987d9c50a2a3f7fe7304f25a45f10ac5626e320ed274fe22257" Nov 03 20:27:12.113904 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:12.113815 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a61961c9210b987d9c50a2a3f7fe7304f25a45f10ac5626e320ed274fe22257"} err="failed to get container status \"3a61961c9210b987d9c50a2a3f7fe7304f25a45f10ac5626e320ed274fe22257\": rpc error: code = NotFound desc = could not find container \"3a61961c9210b987d9c50a2a3f7fe7304f25a45f10ac5626e320ed274fe22257\": container with ID starting with 3a61961c9210b987d9c50a2a3f7fe7304f25a45f10ac5626e320ed274fe22257 not found: ID does not exist" Nov 03 20:27:12.125391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:12.125359 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb"] Nov 03 20:27:12.128564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:12.128525 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b55105-cfb4766d7-gvplb"] Nov 03 20:27:12.184983 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:12.184945 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f695bca-daf7-4b45-baab-e4b6a5c5adbc" path="/var/lib/kubelet/pods/6f695bca-daf7-4b45-baab-e4b6a5c5adbc/volumes" Nov 03 20:27:13.479512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.479484 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:27:13.570978 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.570942 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istiod-ca-cert\") pod \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " Nov 03 20:27:13.571167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.570987 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-workload-certs\") pod \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " Nov 03 20:27:13.571167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.571013 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-credential-socket\") pod \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " Nov 03 20:27:13.571167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.571086 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-workload-socket\") pod \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " Nov 03 20:27:13.571167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.571130 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-envoy\") pod \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " Nov 03 20:27:13.571308 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.571212 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-kube-api-access-b5npf\" (UniqueName: \"kubernetes.io/projected/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-kube-api-access-b5npf\") pod \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " Nov 03 20:27:13.571308 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.571275 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-token\" (UniqueName: \"kubernetes.io/projected/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-token\") pod \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " Nov 03 20:27:13.571378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.571308 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-podinfo\") pod \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " Nov 03 20:27:13.571378 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.571359 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-data\") pod \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\" (UID: \"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f\") " Nov 03 20:27:13.571446 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.571387 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-workload-certs" pod "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" (UID: "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:13.571446 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.571402 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istiod-ca-cert" pod "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" (UID: "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:27:13.571680 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.571654 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-credential-socket" pod "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" (UID: "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:13.571924 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.571663 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-workload-socket" pod "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" (UID: "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:13.571924 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.571816 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:13.571924 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.571913 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:13.572131 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.571947 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-data" pod "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" (UID: "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:13.573839 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.573817 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-token" pod "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" (UID: "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:13.575341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.575318 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-envoy" pod "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" (UID: "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:13.575455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.575321 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-podinfo" pod "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" (UID: "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:27:13.575455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.575354 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-kube-api-access-b5npf" pod "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" (UID: "96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f"). InnerVolumeSpecName "kube-api-access-b5npf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:13.673365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.673330 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:13.673365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.673362 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5npf\" (UniqueName: \"kubernetes.io/projected/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-kube-api-access-b5npf\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:13.673365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.673372 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:13.673638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.673382 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:13.673638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.673391 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:13.673638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.673399 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:13.673638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:13.673407 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:14.114067 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:14.114029 5111 generic.go:356] "Generic (PLEG): container finished" podID="96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" containerID="5af8b136c308d412ed77327969f64975d358c108ce260bed9134c16312cb742f" exitCode=0 Nov 03 20:27:14.114278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:14.114102 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" Nov 03 20:27:14.114278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:14.114109 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" event={"ID":"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f","Type":"ContainerDied","Data":"5af8b136c308d412ed77327969f64975d358c108ce260bed9134c16312cb742f"} Nov 03 20:27:14.114278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:14.114190 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z" event={"ID":"96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f","Type":"ContainerDied","Data":"3f08834b7ed8eff5f27b144e7ea59468d59666422c3a1a15ce499c7029e7a1c8"} Nov 03 20:27:14.114278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:14.114212 5111 scope.go:122] "RemoveContainer" containerID="5af8b136c308d412ed77327969f64975d358c108ce260bed9134c16312cb742f" Nov 03 20:27:14.127199 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:14.127180 5111 scope.go:122] "RemoveContainer" containerID="5af8b136c308d412ed77327969f64975d358c108ce260bed9134c16312cb742f" Nov 03 20:27:14.127589 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:14.127563 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af8b136c308d412ed77327969f64975d358c108ce260bed9134c16312cb742f\": container with ID starting with 5af8b136c308d412ed77327969f64975d358c108ce260bed9134c16312cb742f not found: ID does not exist" containerID="5af8b136c308d412ed77327969f64975d358c108ce260bed9134c16312cb742f" Nov 03 20:27:14.127672 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:14.127601 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af8b136c308d412ed77327969f64975d358c108ce260bed9134c16312cb742f"} err="failed to get container status \"5af8b136c308d412ed77327969f64975d358c108ce260bed9134c16312cb742f\": rpc error: code = NotFound desc = could not find container \"5af8b136c308d412ed77327969f64975d358c108ce260bed9134c16312cb742f\": container with ID starting with 5af8b136c308d412ed77327969f64975d358c108ce260bed9134c16312cb742f not found: ID does not exist" Nov 03 20:27:14.179029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:14.178985 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z"] Nov 03 20:27:14.192491 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:14.192247 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-qcm6z"] Nov 03 20:27:15.087319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.087285 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" podUID="67f55725-b707-46f2-9cf5-8932b47df3b8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:27:15.087319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.087304 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" podUID="8d4c64f2-4b08-4847-ade9-8035b3f7853c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:27:15.175678 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.175637 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8"] Nov 03 20:27:15.175937 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.175902 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerName="istio-proxy" containerID="cri-o://545d1e185fbc992f113b4d2ef69728ab28f974921050e940df2c0971123affca" gracePeriod=30 Nov 03 20:27:15.812828 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.812791 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf"] Nov 03 20:27:15.813309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813292 5111 cpu_manager.go:414] "containerMap: removing container" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" Nov 03 20:27:15.813364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813310 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" Nov 03 20:27:15.813364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813320 5111 cpu_manager.go:414] "containerMap: removing container" podUID="76e0c2bf-68f5-4cbd-a110-c24180dd4525" containerName="revoke-keys" Nov 03 20:27:15.813364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813326 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e0c2bf-68f5-4cbd-a110-c24180dd4525" containerName="revoke-keys" Nov 03 20:27:15.813364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813337 5111 cpu_manager.go:414] "containerMap: removing container" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" Nov 03 20:27:15.813364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813342 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" Nov 03 20:27:15.813364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813351 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6f695bca-daf7-4b45-baab-e4b6a5c5adbc" containerName="payload-pre-processing" Nov 03 20:27:15.813364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813355 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f695bca-daf7-4b45-baab-e4b6a5c5adbc" containerName="payload-pre-processing" Nov 03 20:27:15.813364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813363 5111 cpu_manager.go:414] "containerMap: removing container" podUID="5478802b-2542-47ee-919d-5a8ad79a5e6a" containerName="istio-proxy" Nov 03 20:27:15.813364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813367 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5478802b-2542-47ee-919d-5a8ad79a5e6a" containerName="istio-proxy" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813377 5111 cpu_manager.go:414] "containerMap: removing container" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813381 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813387 5111 cpu_manager.go:414] "containerMap: removing container" podUID="77848c07-7655-4d10-9a19-76e07bc77e8a" containerName="revoke-keys" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813391 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77848c07-7655-4d10-9a19-76e07bc77e8a" containerName="revoke-keys" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813405 5111 cpu_manager.go:414] "containerMap: removing container" podUID="96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" containerName="istio-proxy" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813409 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" containerName="istio-proxy" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813415 5111 cpu_manager.go:414] "containerMap: removing container" podUID="bc60303a-7c35-40e2-a552-4ddf22fa666f" containerName="revoke-keys" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813420 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc60303a-7c35-40e2-a552-4ddf22fa666f" containerName="revoke-keys" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813428 5111 cpu_manager.go:414] "containerMap: removing container" podUID="5478802b-2542-47ee-919d-5a8ad79a5e6a" containerName="istio-proxy" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813432 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5478802b-2542-47ee-919d-5a8ad79a5e6a" containerName="istio-proxy" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813515 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="7959c647-ef80-40d4-8262-335b9e72934b" containerName="main" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813523 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e0c2bf-68f5-4cbd-a110-c24180dd4525" containerName="revoke-keys" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813542 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec0ff9ad-04a1-4709-960f-78e78bbaca58" containerName="main" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813549 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f695bca-daf7-4b45-baab-e4b6a5c5adbc" containerName="payload-pre-processing" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813555 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="5478802b-2542-47ee-919d-5a8ad79a5e6a" containerName="istio-proxy" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813562 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="afbc0e79-dd6e-4c42-9330-ff7988817e80" containerName="main" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813566 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="77848c07-7655-4d10-9a19-76e07bc77e8a" containerName="revoke-keys" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813574 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" containerName="istio-proxy" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813579 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc60303a-7c35-40e2-a552-4ddf22fa666f" containerName="revoke-keys" Nov 03 20:27:15.813660 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.813585 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="5478802b-2542-47ee-919d-5a8ad79a5e6a" containerName="istio-proxy" Nov 03 20:27:15.818654 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.818632 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" Nov 03 20:27:15.827310 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.827279 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf"] Nov 03 20:27:15.843605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.843578 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:27:15.843787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.843711 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:27:15.906329 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.906299 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:27:15.906515 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:15.906411 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:27:16.007696 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:16.007008 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnswp\" (UniqueName: \"kubernetes.io/projected/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-kube-api-access-lnswp\") pod \"maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf\" (UID: \"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" Nov 03 20:27:16.007696 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:16.007165 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf\" (UID: \"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" Nov 03 20:27:16.007696 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:16.007225 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf\" (UID: \"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" Nov 03 20:27:16.108757 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:16.108669 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf\" (UID: \"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" Nov 03 20:27:16.108757 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:16.108734 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf\" (UID: \"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" Nov 03 20:27:16.109204 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:16.108813 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnswp\" (UniqueName: \"kubernetes.io/projected/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-kube-api-access-lnswp\") pod \"maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf\" (UID: \"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" Nov 03 20:27:16.109567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:16.109525 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf\" (UID: \"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" Nov 03 20:27:16.111376 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:16.111341 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf\" (UID: \"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" Nov 03 20:27:16.117277 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:16.117246 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnswp\" (UniqueName: \"kubernetes.io/projected/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-kube-api-access-lnswp\") pod \"maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf\" (UID: \"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" Nov 03 20:27:16.143240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:16.143211 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" Nov 03 20:27:16.183784 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:16.183742 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f" path="/var/lib/kubelet/pods/96cc3ca7-a54e-4172-92aa-3ffe98ee0b1f/volumes" Nov 03 20:27:16.433978 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:16.433954 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf"] Nov 03 20:27:16.436071 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:27:16.436031 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed18a08_f3d1_456c_a6a2_9b73f5eb866a.slice/crio-230c346faedcb61938a76989d72003ad138502ec144f9df22548ae3f1c133dd4 WatchSource:0}: Error finding container 230c346faedcb61938a76989d72003ad138502ec144f9df22548ae3f1c133dd4: Status 404 returned error can't find the container with id 230c346faedcb61938a76989d72003ad138502ec144f9df22548ae3f1c133dd4 Nov 03 20:27:17.144368 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:17.144328 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" event={"ID":"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a","Type":"ContainerStarted","Data":"c24656c020f30c09ad0a7d1fbb18af7e9274dde774b1ac3fe155ad246d05d29d"} Nov 03 20:27:17.144969 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:17.144397 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" event={"ID":"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a","Type":"ContainerStarted","Data":"230c346faedcb61938a76989d72003ad138502ec144f9df22548ae3f1c133dd4"} Nov 03 20:27:17.167605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:17.167551 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" podStartSLOduration=2.167519848 podStartE2EDuration="2.167519848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:17.165588783 +0000 UTC m=+2867.127967221" watchObservedRunningTime="2026-11-03 20:27:17.167519848 +0000 UTC m=+2867.129898286" Nov 03 20:27:17.389833 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:17.389803 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh"] Nov 03 20:27:17.390039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:17.390012 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" podUID="ff1c43d3-728e-4c45-a379-d3ea36351718" containerName="istio-proxy" containerID="cri-o://3437d93556a9297a6648a0d5a825c01049bc8dc7be9eb2da1998aa0b83b83765" gracePeriod=30 Nov 03 20:27:17.842214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:17.842131 5111 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:27:17.842214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:17.842199 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" podUID="ff1c43d3-728e-4c45-a379-d3ea36351718" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:27:18.153895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:18.153860 5111 generic.go:356] "Generic (PLEG): container finished" podID="0ed18a08-f3d1-456c-a6a2-9b73f5eb866a" containerID="c24656c020f30c09ad0a7d1fbb18af7e9274dde774b1ac3fe155ad246d05d29d" exitCode=0 Nov 03 20:27:18.154318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:18.153906 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" event={"ID":"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a","Type":"ContainerDied","Data":"c24656c020f30c09ad0a7d1fbb18af7e9274dde774b1ac3fe155ad246d05d29d"} Nov 03 20:27:18.282390 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:18.282342 5111 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:27:18.282597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:18.282414 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:27:18.676411 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:18.676344 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" Nov 03 20:27:18.745078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:18.745043 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f55725-b707-46f2-9cf5-8932b47df3b8-kube-api-access-nbg5j\" (UniqueName: \"kubernetes.io/projected/67f55725-b707-46f2-9cf5-8932b47df3b8-kube-api-access-nbg5j\") pod \"67f55725-b707-46f2-9cf5-8932b47df3b8\" (UID: \"67f55725-b707-46f2-9cf5-8932b47df3b8\") " Nov 03 20:27:18.745283 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:18.745204 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67f55725-b707-46f2-9cf5-8932b47df3b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f55725-b707-46f2-9cf5-8932b47df3b8-plugins-config-volume\") pod \"67f55725-b707-46f2-9cf5-8932b47df3b8\" (UID: \"67f55725-b707-46f2-9cf5-8932b47df3b8\") " Nov 03 20:27:18.746655 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:18.746627 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f55725-b707-46f2-9cf5-8932b47df3b8-plugins-config-volume" pod "67f55725-b707-46f2-9cf5-8932b47df3b8" (UID: "67f55725-b707-46f2-9cf5-8932b47df3b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:27:18.747365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:18.747336 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f55725-b707-46f2-9cf5-8932b47df3b8-kube-api-access-nbg5j" pod "67f55725-b707-46f2-9cf5-8932b47df3b8" (UID: "67f55725-b707-46f2-9cf5-8932b47df3b8"). InnerVolumeSpecName "kube-api-access-nbg5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:18.846540 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:18.846505 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbg5j\" (UniqueName: \"kubernetes.io/projected/67f55725-b707-46f2-9cf5-8932b47df3b8-kube-api-access-nbg5j\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:18.846726 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:18.846554 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f55725-b707-46f2-9cf5-8932b47df3b8-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:18.998336 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:18.998249 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw"] Nov 03 20:27:18.999032 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:18.998448 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" podUID="eea4d795-91eb-4eed-9ce9-891d4d26337c" containerName="istio-proxy" containerID="cri-o://c5234fa8702820db67bec68464d179e92f8b9eae792cb8795e5f71126c9ccfbf" gracePeriod=30 Nov 03 20:27:19.163019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.162981 5111 generic.go:356] "Generic (PLEG): container finished" podID="67f55725-b707-46f2-9cf5-8932b47df3b8" containerID="8306cc4bbe5d8c8e75626e264e1ab35c1dac9aa40f6cface6df09adedd14978a" exitCode=0 Nov 03 20:27:19.163512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.163029 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" Nov 03 20:27:19.163512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.163065 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" event={"ID":"67f55725-b707-46f2-9cf5-8932b47df3b8","Type":"ContainerDied","Data":"8306cc4bbe5d8c8e75626e264e1ab35c1dac9aa40f6cface6df09adedd14978a"} Nov 03 20:27:19.163512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.163119 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6" event={"ID":"67f55725-b707-46f2-9cf5-8932b47df3b8","Type":"ContainerDied","Data":"4d8d5230cb41f2082eab23bda1f636c298356a2228c6b14b08fd2fab39d2bd1c"} Nov 03 20:27:19.163512 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.163140 5111 scope.go:122] "RemoveContainer" containerID="8306cc4bbe5d8c8e75626e264e1ab35c1dac9aa40f6cface6df09adedd14978a" Nov 03 20:27:19.176843 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.176824 5111 scope.go:122] "RemoveContainer" containerID="8306cc4bbe5d8c8e75626e264e1ab35c1dac9aa40f6cface6df09adedd14978a" Nov 03 20:27:19.177181 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:19.177113 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8306cc4bbe5d8c8e75626e264e1ab35c1dac9aa40f6cface6df09adedd14978a\": container with ID starting with 8306cc4bbe5d8c8e75626e264e1ab35c1dac9aa40f6cface6df09adedd14978a not found: ID does not exist" containerID="8306cc4bbe5d8c8e75626e264e1ab35c1dac9aa40f6cface6df09adedd14978a" Nov 03 20:27:19.177274 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.177200 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8306cc4bbe5d8c8e75626e264e1ab35c1dac9aa40f6cface6df09adedd14978a"} err="failed to get container status \"8306cc4bbe5d8c8e75626e264e1ab35c1dac9aa40f6cface6df09adedd14978a\": rpc error: code = NotFound desc = could not find container \"8306cc4bbe5d8c8e75626e264e1ab35c1dac9aa40f6cface6df09adedd14978a\": container with ID starting with 8306cc4bbe5d8c8e75626e264e1ab35c1dac9aa40f6cface6df09adedd14978a not found: ID does not exist" Nov 03 20:27:19.195370 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.195336 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6"] Nov 03 20:27:19.200061 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.200033 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ab1e46-57c8758d5d-gzwb6"] Nov 03 20:27:19.297712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.297683 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" Nov 03 20:27:19.352633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.352600 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-kube-api-access-lnswp\" (UniqueName: \"kubernetes.io/projected/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-kube-api-access-lnswp\") pod \"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a\" (UID: \"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a\") " Nov 03 20:27:19.352806 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.352663 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-maas-api-openshift-ca\") pod \"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a\" (UID: \"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a\") " Nov 03 20:27:19.352806 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.352761 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-maas-api-xks-ca\") pod \"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a\" (UID: \"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a\") " Nov 03 20:27:19.353065 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.353027 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-maas-api-openshift-ca" pod "0ed18a08-f3d1-456c-a6a2-9b73f5eb866a" (UID: "0ed18a08-f3d1-456c-a6a2-9b73f5eb866a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:27:19.354799 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.354777 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-maas-api-xks-ca" pod "0ed18a08-f3d1-456c-a6a2-9b73f5eb866a" (UID: "0ed18a08-f3d1-456c-a6a2-9b73f5eb866a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:27:19.354925 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.354907 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-kube-api-access-lnswp" pod "0ed18a08-f3d1-456c-a6a2-9b73f5eb866a" (UID: "0ed18a08-f3d1-456c-a6a2-9b73f5eb866a"). InnerVolumeSpecName "kube-api-access-lnswp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:19.454701 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.454654 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnswp\" (UniqueName: \"kubernetes.io/projected/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-kube-api-access-lnswp\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:19.454701 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.454696 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:19.454981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:19.454712 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:20.171939 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.171904 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" Nov 03 20:27:20.172410 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.171956 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf" event={"ID":"0ed18a08-f3d1-456c-a6a2-9b73f5eb866a","Type":"ContainerDied","Data":"230c346faedcb61938a76989d72003ad138502ec144f9df22548ae3f1c133dd4"} Nov 03 20:27:20.172410 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.172012 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="230c346faedcb61938a76989d72003ad138502ec144f9df22548ae3f1c133dd4" Nov 03 20:27:20.182880 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.182849 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67f55725-b707-46f2-9cf5-8932b47df3b8" path="/var/lib/kubelet/pods/67f55725-b707-46f2-9cf5-8932b47df3b8/volumes" Nov 03 20:27:20.460584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.460524 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" Nov 03 20:27:20.502617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.502584 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:27:20.570678 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.570642 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-workload-socket\") pod \"2815d45f-e02c-4178-9f95-0375fda5f11c\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " Nov 03 20:27:20.570895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.570698 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d4c64f2-4b08-4847-ade9-8035b3f7853c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4c64f2-4b08-4847-ade9-8035b3f7853c-plugins-config-volume\") pod \"8d4c64f2-4b08-4847-ade9-8035b3f7853c\" (UID: \"8d4c64f2-4b08-4847-ade9-8035b3f7853c\") " Nov 03 20:27:20.570895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.570719 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2815d45f-e02c-4178-9f95-0375fda5f11c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2815d45f-e02c-4178-9f95-0375fda5f11c-istiod-ca-cert\") pod \"2815d45f-e02c-4178-9f95-0375fda5f11c\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " Nov 03 20:27:20.570895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.570751 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-token\" (UniqueName: \"kubernetes.io/projected/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-token\") pod \"2815d45f-e02c-4178-9f95-0375fda5f11c\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " Nov 03 20:27:20.570895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.570792 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2815d45f-e02c-4178-9f95-0375fda5f11c-kube-api-access-n6gmh\" (UniqueName: \"kubernetes.io/projected/2815d45f-e02c-4178-9f95-0375fda5f11c-kube-api-access-n6gmh\") pod \"2815d45f-e02c-4178-9f95-0375fda5f11c\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " Nov 03 20:27:20.570895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.570828 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-envoy\") pod \"2815d45f-e02c-4178-9f95-0375fda5f11c\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " Nov 03 20:27:20.570895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.570888 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4c64f2-4b08-4847-ade9-8035b3f7853c-kube-api-access-9n8pn\" (UniqueName: \"kubernetes.io/projected/8d4c64f2-4b08-4847-ade9-8035b3f7853c-kube-api-access-9n8pn\") pod \"8d4c64f2-4b08-4847-ade9-8035b3f7853c\" (UID: \"8d4c64f2-4b08-4847-ade9-8035b3f7853c\") " Nov 03 20:27:20.571291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.570947 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-workload-certs\") pod \"2815d45f-e02c-4178-9f95-0375fda5f11c\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " Nov 03 20:27:20.571291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.570992 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-workload-socket" pod "2815d45f-e02c-4178-9f95-0375fda5f11c" (UID: "2815d45f-e02c-4178-9f95-0375fda5f11c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:20.571291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.571048 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-podinfo\") pod \"2815d45f-e02c-4178-9f95-0375fda5f11c\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " Nov 03 20:27:20.571291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.571196 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-credential-socket\") pod \"2815d45f-e02c-4178-9f95-0375fda5f11c\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " Nov 03 20:27:20.571291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.571245 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-data\") pod \"2815d45f-e02c-4178-9f95-0375fda5f11c\" (UID: \"2815d45f-e02c-4178-9f95-0375fda5f11c\") " Nov 03 20:27:20.571588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.571290 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-workload-certs" pod "2815d45f-e02c-4178-9f95-0375fda5f11c" (UID: "2815d45f-e02c-4178-9f95-0375fda5f11c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:20.571588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.571213 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2815d45f-e02c-4178-9f95-0375fda5f11c-istiod-ca-cert" pod "2815d45f-e02c-4178-9f95-0375fda5f11c" (UID: "2815d45f-e02c-4178-9f95-0375fda5f11c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:27:20.571588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.571497 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-credential-socket" pod "2815d45f-e02c-4178-9f95-0375fda5f11c" (UID: "2815d45f-e02c-4178-9f95-0375fda5f11c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:20.571754 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.571656 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-data" pod "2815d45f-e02c-4178-9f95-0375fda5f11c" (UID: "2815d45f-e02c-4178-9f95-0375fda5f11c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:20.571809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.571769 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:20.571809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.571787 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:20.571930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.571801 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:20.571930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.571843 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:20.571930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.571858 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2815d45f-e02c-4178-9f95-0375fda5f11c-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:20.572098 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.571990 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4c64f2-4b08-4847-ade9-8035b3f7853c-plugins-config-volume" pod "8d4c64f2-4b08-4847-ade9-8035b3f7853c" (UID: "8d4c64f2-4b08-4847-ade9-8035b3f7853c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:27:20.573488 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.573466 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4c64f2-4b08-4847-ade9-8035b3f7853c-kube-api-access-9n8pn" pod "8d4c64f2-4b08-4847-ade9-8035b3f7853c" (UID: "8d4c64f2-4b08-4847-ade9-8035b3f7853c"). InnerVolumeSpecName "kube-api-access-9n8pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:20.573782 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.573763 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2815d45f-e02c-4178-9f95-0375fda5f11c-kube-api-access-n6gmh" pod "2815d45f-e02c-4178-9f95-0375fda5f11c" (UID: "2815d45f-e02c-4178-9f95-0375fda5f11c"). InnerVolumeSpecName "kube-api-access-n6gmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:20.574078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.574051 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-envoy" pod "2815d45f-e02c-4178-9f95-0375fda5f11c" (UID: "2815d45f-e02c-4178-9f95-0375fda5f11c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:20.574191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.574153 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-podinfo" pod "2815d45f-e02c-4178-9f95-0375fda5f11c" (UID: "2815d45f-e02c-4178-9f95-0375fda5f11c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:27:20.574191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.574177 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-token" pod "2815d45f-e02c-4178-9f95-0375fda5f11c" (UID: "2815d45f-e02c-4178-9f95-0375fda5f11c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:20.622813 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.622773 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65"] Nov 03 20:27:20.637031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.636998 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks"] Nov 03 20:27:20.647308 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.647280 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm"] Nov 03 20:27:20.673072 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.673042 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:20.673072 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.673072 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6gmh\" (UniqueName: \"kubernetes.io/projected/2815d45f-e02c-4178-9f95-0375fda5f11c-kube-api-access-n6gmh\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:20.673285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.673084 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:20.673285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.673094 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n8pn\" (UniqueName: \"kubernetes.io/projected/8d4c64f2-4b08-4847-ade9-8035b3f7853c-kube-api-access-9n8pn\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:20.673285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.673105 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2815d45f-e02c-4178-9f95-0375fda5f11c-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:20.673285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:20.673113 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4c64f2-4b08-4847-ade9-8035b3f7853c-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:21.182315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.182281 5111 generic.go:356] "Generic (PLEG): container finished" podID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerID="545d1e185fbc992f113b4d2ef69728ab28f974921050e940df2c0971123affca" exitCode=0 Nov 03 20:27:21.182830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.182339 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" Nov 03 20:27:21.182830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.182366 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" event={"ID":"2815d45f-e02c-4178-9f95-0375fda5f11c","Type":"ContainerDied","Data":"545d1e185fbc992f113b4d2ef69728ab28f974921050e940df2c0971123affca"} Nov 03 20:27:21.182830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.182419 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8" event={"ID":"2815d45f-e02c-4178-9f95-0375fda5f11c","Type":"ContainerDied","Data":"826f0283964bc89f6089a897d6fef0a4248efbfa93095eac2661ab7c02ca07ad"} Nov 03 20:27:21.182830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.182445 5111 scope.go:122] "RemoveContainer" containerID="545d1e185fbc992f113b4d2ef69728ab28f974921050e940df2c0971123affca" Nov 03 20:27:21.184005 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.183976 5111 generic.go:356] "Generic (PLEG): container finished" podID="8d4c64f2-4b08-4847-ade9-8035b3f7853c" containerID="a73901a8b96750605632061a49b742f4d6e607b3571d12dd648a954dbb923474" exitCode=0 Nov 03 20:27:21.184133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.184044 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" Nov 03 20:27:21.184133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.184048 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" event={"ID":"8d4c64f2-4b08-4847-ade9-8035b3f7853c","Type":"ContainerDied","Data":"a73901a8b96750605632061a49b742f4d6e607b3571d12dd648a954dbb923474"} Nov 03 20:27:21.184133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.184101 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j" event={"ID":"8d4c64f2-4b08-4847-ade9-8035b3f7853c","Type":"ContainerDied","Data":"330ffae8d76b91261c586f5500b249f6ebcc7831923b010d856935bf0fb4e5c9"} Nov 03 20:27:21.195293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.195254 5111 scope.go:122] "RemoveContainer" containerID="f237b247cefe086452423efa17b16ea60f036937087d5c6d50da4f16ebb0448b" Nov 03 20:27:21.205728 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.205701 5111 scope.go:122] "RemoveContainer" containerID="545d1e185fbc992f113b4d2ef69728ab28f974921050e940df2c0971123affca" Nov 03 20:27:21.206003 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:21.205983 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545d1e185fbc992f113b4d2ef69728ab28f974921050e940df2c0971123affca\": container with ID starting with 545d1e185fbc992f113b4d2ef69728ab28f974921050e940df2c0971123affca not found: ID does not exist" containerID="545d1e185fbc992f113b4d2ef69728ab28f974921050e940df2c0971123affca" Nov 03 20:27:21.206062 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.206016 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545d1e185fbc992f113b4d2ef69728ab28f974921050e940df2c0971123affca"} err="failed to get container status \"545d1e185fbc992f113b4d2ef69728ab28f974921050e940df2c0971123affca\": rpc error: code = NotFound desc = could not find container \"545d1e185fbc992f113b4d2ef69728ab28f974921050e940df2c0971123affca\": container with ID starting with 545d1e185fbc992f113b4d2ef69728ab28f974921050e940df2c0971123affca not found: ID does not exist" Nov 03 20:27:21.206062 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.206037 5111 scope.go:122] "RemoveContainer" containerID="f237b247cefe086452423efa17b16ea60f036937087d5c6d50da4f16ebb0448b" Nov 03 20:27:21.206314 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:21.206300 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f237b247cefe086452423efa17b16ea60f036937087d5c6d50da4f16ebb0448b\": container with ID starting with f237b247cefe086452423efa17b16ea60f036937087d5c6d50da4f16ebb0448b not found: ID does not exist" containerID="f237b247cefe086452423efa17b16ea60f036937087d5c6d50da4f16ebb0448b" Nov 03 20:27:21.206368 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.206318 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f237b247cefe086452423efa17b16ea60f036937087d5c6d50da4f16ebb0448b"} err="failed to get container status \"f237b247cefe086452423efa17b16ea60f036937087d5c6d50da4f16ebb0448b\": rpc error: code = NotFound desc = could not find container \"f237b247cefe086452423efa17b16ea60f036937087d5c6d50da4f16ebb0448b\": container with ID starting with f237b247cefe086452423efa17b16ea60f036937087d5c6d50da4f16ebb0448b not found: ID does not exist" Nov 03 20:27:21.206368 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.206330 5111 scope.go:122] "RemoveContainer" containerID="a73901a8b96750605632061a49b742f4d6e607b3571d12dd648a954dbb923474" Nov 03 20:27:21.209295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.209274 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8"] Nov 03 20:27:21.216864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.216844 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-4h4w8"] Nov 03 20:27:21.220979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.220962 5111 scope.go:122] "RemoveContainer" containerID="a73901a8b96750605632061a49b742f4d6e607b3571d12dd648a954dbb923474" Nov 03 20:27:21.221230 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:21.221210 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73901a8b96750605632061a49b742f4d6e607b3571d12dd648a954dbb923474\": container with ID starting with a73901a8b96750605632061a49b742f4d6e607b3571d12dd648a954dbb923474 not found: ID does not exist" containerID="a73901a8b96750605632061a49b742f4d6e607b3571d12dd648a954dbb923474" Nov 03 20:27:21.221295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.221237 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73901a8b96750605632061a49b742f4d6e607b3571d12dd648a954dbb923474"} err="failed to get container status \"a73901a8b96750605632061a49b742f4d6e607b3571d12dd648a954dbb923474\": rpc error: code = NotFound desc = could not find container \"a73901a8b96750605632061a49b742f4d6e607b3571d12dd648a954dbb923474\": container with ID starting with a73901a8b96750605632061a49b742f4d6e607b3571d12dd648a954dbb923474 not found: ID does not exist" Nov 03 20:27:21.232871 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.232831 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j"] Nov 03 20:27:21.238859 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:21.238820 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b42aea-65b5c44c55-zgh9j"] Nov 03 20:27:22.181607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.181522 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" path="/var/lib/kubelet/pods/2815d45f-e02c-4178-9f95-0375fda5f11c/volumes" Nov 03 20:27:22.182118 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.182100 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4c64f2-4b08-4847-ade9-8035b3f7853c" path="/var/lib/kubelet/pods/8d4c64f2-4b08-4847-ade9-8035b3f7853c/volumes" Nov 03 20:27:22.665758 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.665735 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:27:22.697688 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.697643 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff1c43d3-728e-4c45-a379-d3ea36351718-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1c43d3-728e-4c45-a379-d3ea36351718-istiod-ca-cert\") pod \"ff1c43d3-728e-4c45-a379-d3ea36351718\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " Nov 03 20:27:22.697688 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.697703 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-workload-socket\") pod \"ff1c43d3-728e-4c45-a379-d3ea36351718\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " Nov 03 20:27:22.697972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.697746 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-envoy\") pod \"ff1c43d3-728e-4c45-a379-d3ea36351718\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " Nov 03 20:27:22.697972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.697837 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1c43d3-728e-4c45-a379-d3ea36351718-kube-api-access-rsvg2\" (UniqueName: \"kubernetes.io/projected/ff1c43d3-728e-4c45-a379-d3ea36351718-kube-api-access-rsvg2\") pod \"ff1c43d3-728e-4c45-a379-d3ea36351718\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " Nov 03 20:27:22.697972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.697897 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-workload-certs\") pod \"ff1c43d3-728e-4c45-a379-d3ea36351718\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " Nov 03 20:27:22.697972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.697948 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-data\") pod \"ff1c43d3-728e-4c45-a379-d3ea36351718\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " Nov 03 20:27:22.698195 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.697995 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-token\" (UniqueName: \"kubernetes.io/projected/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-token\") pod \"ff1c43d3-728e-4c45-a379-d3ea36351718\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " Nov 03 20:27:22.698195 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.698027 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-credential-socket\") pod \"ff1c43d3-728e-4c45-a379-d3ea36351718\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " Nov 03 20:27:22.698195 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.698094 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-podinfo\") pod \"ff1c43d3-728e-4c45-a379-d3ea36351718\" (UID: \"ff1c43d3-728e-4c45-a379-d3ea36351718\") " Nov 03 20:27:22.698604 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.698093 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-workload-socket" pod "ff1c43d3-728e-4c45-a379-d3ea36351718" (UID: "ff1c43d3-728e-4c45-a379-d3ea36351718"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:22.698604 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.698099 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1c43d3-728e-4c45-a379-d3ea36351718-istiod-ca-cert" pod "ff1c43d3-728e-4c45-a379-d3ea36351718" (UID: "ff1c43d3-728e-4c45-a379-d3ea36351718"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:27:22.698604 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.698309 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-workload-certs" pod "ff1c43d3-728e-4c45-a379-d3ea36351718" (UID: "ff1c43d3-728e-4c45-a379-d3ea36351718"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:22.699200 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.698864 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-credential-socket" pod "ff1c43d3-728e-4c45-a379-d3ea36351718" (UID: "ff1c43d3-728e-4c45-a379-d3ea36351718"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:22.699200 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.699144 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-data" pod "ff1c43d3-728e-4c45-a379-d3ea36351718" (UID: "ff1c43d3-728e-4c45-a379-d3ea36351718"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:22.700691 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.700659 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-podinfo" pod "ff1c43d3-728e-4c45-a379-d3ea36351718" (UID: "ff1c43d3-728e-4c45-a379-d3ea36351718"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:27:22.700812 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.700718 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1c43d3-728e-4c45-a379-d3ea36351718-kube-api-access-rsvg2" pod "ff1c43d3-728e-4c45-a379-d3ea36351718" (UID: "ff1c43d3-728e-4c45-a379-d3ea36351718"). InnerVolumeSpecName "kube-api-access-rsvg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:22.700880 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.700811 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-envoy" pod "ff1c43d3-728e-4c45-a379-d3ea36351718" (UID: "ff1c43d3-728e-4c45-a379-d3ea36351718"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:22.700943 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.700890 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-token" pod "ff1c43d3-728e-4c45-a379-d3ea36351718" (UID: "ff1c43d3-728e-4c45-a379-d3ea36351718"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:22.799526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.799441 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1c43d3-728e-4c45-a379-d3ea36351718-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:22.799526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.799470 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:22.799526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.799483 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:22.799526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.799495 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsvg2\" (UniqueName: \"kubernetes.io/projected/ff1c43d3-728e-4c45-a379-d3ea36351718-kube-api-access-rsvg2\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:22.799526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.799508 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:22.799526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.799520 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:22.799875 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.799565 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:22.799875 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.799575 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1c43d3-728e-4c45-a379-d3ea36351718-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:22.799875 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:22.799584 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1c43d3-728e-4c45-a379-d3ea36351718-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:23.201899 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:23.201863 5111 generic.go:356] "Generic (PLEG): container finished" podID="ff1c43d3-728e-4c45-a379-d3ea36351718" containerID="3437d93556a9297a6648a0d5a825c01049bc8dc7be9eb2da1998aa0b83b83765" exitCode=0 Nov 03 20:27:23.202119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:23.201921 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" Nov 03 20:27:23.202119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:23.201946 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" event={"ID":"ff1c43d3-728e-4c45-a379-d3ea36351718","Type":"ContainerDied","Data":"3437d93556a9297a6648a0d5a825c01049bc8dc7be9eb2da1998aa0b83b83765"} Nov 03 20:27:23.202119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:23.202005 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh" event={"ID":"ff1c43d3-728e-4c45-a379-d3ea36351718","Type":"ContainerDied","Data":"9d6cb6e94aae8d9666a48e2502e49df679becf32c92731feaabf21fd16696427"} Nov 03 20:27:23.202119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:23.202028 5111 scope.go:122] "RemoveContainer" containerID="3437d93556a9297a6648a0d5a825c01049bc8dc7be9eb2da1998aa0b83b83765" Nov 03 20:27:23.214221 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:23.214196 5111 scope.go:122] "RemoveContainer" containerID="3437d93556a9297a6648a0d5a825c01049bc8dc7be9eb2da1998aa0b83b83765" Nov 03 20:27:23.214526 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:23.214502 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3437d93556a9297a6648a0d5a825c01049bc8dc7be9eb2da1998aa0b83b83765\": container with ID starting with 3437d93556a9297a6648a0d5a825c01049bc8dc7be9eb2da1998aa0b83b83765 not found: ID does not exist" containerID="3437d93556a9297a6648a0d5a825c01049bc8dc7be9eb2da1998aa0b83b83765" Nov 03 20:27:23.214637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:23.214550 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3437d93556a9297a6648a0d5a825c01049bc8dc7be9eb2da1998aa0b83b83765"} err="failed to get container status \"3437d93556a9297a6648a0d5a825c01049bc8dc7be9eb2da1998aa0b83b83765\": rpc error: code = NotFound desc = could not find container \"3437d93556a9297a6648a0d5a825c01049bc8dc7be9eb2da1998aa0b83b83765\": container with ID starting with 3437d93556a9297a6648a0d5a825c01049bc8dc7be9eb2da1998aa0b83b83765 not found: ID does not exist" Nov 03 20:27:23.226844 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:23.226816 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh"] Nov 03 20:27:23.233100 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:23.233074 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-ss7mh"] Nov 03 20:27:24.185950 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.185917 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1c43d3-728e-4c45-a379-d3ea36351718" path="/var/lib/kubelet/pods/ff1c43d3-728e-4c45-a379-d3ea36351718/volumes" Nov 03 20:27:24.216596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.216564 5111 generic.go:356] "Generic (PLEG): container finished" podID="eea4d795-91eb-4eed-9ce9-891d4d26337c" containerID="c5234fa8702820db67bec68464d179e92f8b9eae792cb8795e5f71126c9ccfbf" exitCode=0 Nov 03 20:27:24.216815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.216645 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" event={"ID":"eea4d795-91eb-4eed-9ce9-891d4d26337c","Type":"ContainerDied","Data":"c5234fa8702820db67bec68464d179e92f8b9eae792cb8795e5f71126c9ccfbf"} Nov 03 20:27:24.261485 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.261453 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:27:24.313764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.313669 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea4d795-91eb-4eed-9ce9-891d4d26337c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea4d795-91eb-4eed-9ce9-891d4d26337c-istiod-ca-cert\") pod \"eea4d795-91eb-4eed-9ce9-891d4d26337c\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " Nov 03 20:27:24.313953 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.313767 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-credential-socket\") pod \"eea4d795-91eb-4eed-9ce9-891d4d26337c\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " Nov 03 20:27:24.313953 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.313808 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-workload-socket\") pod \"eea4d795-91eb-4eed-9ce9-891d4d26337c\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " Nov 03 20:27:24.313953 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.313898 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-token\" (UniqueName: \"kubernetes.io/projected/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-token\") pod \"eea4d795-91eb-4eed-9ce9-891d4d26337c\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " Nov 03 20:27:24.314124 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.313971 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-workload-certs\") pod \"eea4d795-91eb-4eed-9ce9-891d4d26337c\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " Nov 03 20:27:24.314124 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.314061 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea4d795-91eb-4eed-9ce9-891d4d26337c-kube-api-access-45jhd\" (UniqueName: \"kubernetes.io/projected/eea4d795-91eb-4eed-9ce9-891d4d26337c-kube-api-access-45jhd\") pod \"eea4d795-91eb-4eed-9ce9-891d4d26337c\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " Nov 03 20:27:24.314124 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.314093 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-podinfo\") pod \"eea4d795-91eb-4eed-9ce9-891d4d26337c\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " Nov 03 20:27:24.314332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.314132 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-data\") pod \"eea4d795-91eb-4eed-9ce9-891d4d26337c\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " Nov 03 20:27:24.314332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.314169 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-envoy\") pod \"eea4d795-91eb-4eed-9ce9-891d4d26337c\" (UID: \"eea4d795-91eb-4eed-9ce9-891d4d26337c\") " Nov 03 20:27:24.316366 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.315896 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-credential-socket" pod "eea4d795-91eb-4eed-9ce9-891d4d26337c" (UID: "eea4d795-91eb-4eed-9ce9-891d4d26337c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:24.316366 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.316044 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea4d795-91eb-4eed-9ce9-891d4d26337c-istiod-ca-cert" pod "eea4d795-91eb-4eed-9ce9-891d4d26337c" (UID: "eea4d795-91eb-4eed-9ce9-891d4d26337c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:27:24.316366 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.316126 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-workload-socket" pod "eea4d795-91eb-4eed-9ce9-891d4d26337c" (UID: "eea4d795-91eb-4eed-9ce9-891d4d26337c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:24.316668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.316525 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-data" pod "eea4d795-91eb-4eed-9ce9-891d4d26337c" (UID: "eea4d795-91eb-4eed-9ce9-891d4d26337c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:24.317007 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.316977 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-workload-certs" pod "eea4d795-91eb-4eed-9ce9-891d4d26337c" (UID: "eea4d795-91eb-4eed-9ce9-891d4d26337c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:24.319130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.319092 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-token" pod "eea4d795-91eb-4eed-9ce9-891d4d26337c" (UID: "eea4d795-91eb-4eed-9ce9-891d4d26337c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:24.325232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.321429 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea4d795-91eb-4eed-9ce9-891d4d26337c-kube-api-access-45jhd" pod "eea4d795-91eb-4eed-9ce9-891d4d26337c" (UID: "eea4d795-91eb-4eed-9ce9-891d4d26337c"). InnerVolumeSpecName "kube-api-access-45jhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:24.325232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.323354 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-podinfo" pod "eea4d795-91eb-4eed-9ce9-891d4d26337c" (UID: "eea4d795-91eb-4eed-9ce9-891d4d26337c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:27:24.332369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.332324 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-envoy" pod "eea4d795-91eb-4eed-9ce9-891d4d26337c" (UID: "eea4d795-91eb-4eed-9ce9-891d4d26337c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:24.416108 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.416064 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea4d795-91eb-4eed-9ce9-891d4d26337c-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:24.416108 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.416100 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:24.416108 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.416109 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:24.416108 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.416118 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:24.416401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.416128 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:24.416401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.416137 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45jhd\" (UniqueName: \"kubernetes.io/projected/eea4d795-91eb-4eed-9ce9-891d4d26337c-kube-api-access-45jhd\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:24.416401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.416146 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:24.416401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.416154 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:24.416401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:24.416163 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea4d795-91eb-4eed-9ce9-891d4d26337c-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:25.225732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:25.225698 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" Nov 03 20:27:25.225732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:25.225722 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw" event={"ID":"eea4d795-91eb-4eed-9ce9-891d4d26337c","Type":"ContainerDied","Data":"b7b23099cd2238bdbd7f09cfac84cce9b8896838d09484f7cc7c0c525d19f715"} Nov 03 20:27:25.226319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:25.225793 5111 scope.go:122] "RemoveContainer" containerID="c5234fa8702820db67bec68464d179e92f8b9eae792cb8795e5f71126c9ccfbf" Nov 03 20:27:25.270370 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:25.252772 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw"] Nov 03 20:27:25.270370 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:25.258828 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-xr8cw"] Nov 03 20:27:26.199579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.199522 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea4d795-91eb-4eed-9ce9-891d4d26337c" path="/var/lib/kubelet/pods/eea4d795-91eb-4eed-9ce9-891d4d26337c/volumes" Nov 03 20:27:26.544082 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.544003 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq"] Nov 03 20:27:26.544691 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.544670 5111 cpu_manager.go:414] "containerMap: removing container" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerName="istio-proxy" Nov 03 20:27:26.544798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.544693 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerName="istio-proxy" Nov 03 20:27:26.544798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.544711 5111 cpu_manager.go:414] "containerMap: removing container" podUID="67f55725-b707-46f2-9cf5-8932b47df3b8" containerName="payload-processing" Nov 03 20:27:26.544798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.544719 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f55725-b707-46f2-9cf5-8932b47df3b8" containerName="payload-processing" Nov 03 20:27:26.544798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.544752 5111 cpu_manager.go:414] "containerMap: removing container" podUID="0ed18a08-f3d1-456c-a6a2-9b73f5eb866a" containerName="revoke-keys" Nov 03 20:27:26.544798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.544760 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed18a08-f3d1-456c-a6a2-9b73f5eb866a" containerName="revoke-keys" Nov 03 20:27:26.544798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.544787 5111 cpu_manager.go:414] "containerMap: removing container" podUID="ff1c43d3-728e-4c45-a379-d3ea36351718" containerName="istio-proxy" Nov 03 20:27:26.544798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.544794 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1c43d3-728e-4c45-a379-d3ea36351718" containerName="istio-proxy" Nov 03 20:27:26.545236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.544812 5111 cpu_manager.go:414] "containerMap: removing container" podUID="eea4d795-91eb-4eed-9ce9-891d4d26337c" containerName="istio-proxy" Nov 03 20:27:26.545236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.544820 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea4d795-91eb-4eed-9ce9-891d4d26337c" containerName="istio-proxy" Nov 03 20:27:26.545236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.544831 5111 cpu_manager.go:414] "containerMap: removing container" podUID="8d4c64f2-4b08-4847-ade9-8035b3f7853c" containerName="payload-pre-processing" Nov 03 20:27:26.545236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.544839 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4c64f2-4b08-4847-ade9-8035b3f7853c" containerName="payload-pre-processing" Nov 03 20:27:26.545236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.544970 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed18a08-f3d1-456c-a6a2-9b73f5eb866a" containerName="revoke-keys" Nov 03 20:27:26.545236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.544993 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1c43d3-728e-4c45-a379-d3ea36351718" containerName="istio-proxy" Nov 03 20:27:26.545236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.545004 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea4d795-91eb-4eed-9ce9-891d4d26337c" containerName="istio-proxy" Nov 03 20:27:26.545236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.545012 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4c64f2-4b08-4847-ade9-8035b3f7853c" containerName="payload-pre-processing" Nov 03 20:27:26.545236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.545024 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerName="istio-proxy" Nov 03 20:27:26.545236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.545036 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerName="istio-proxy" Nov 03 20:27:26.545236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.545046 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f55725-b707-46f2-9cf5-8932b47df3b8" containerName="payload-processing" Nov 03 20:27:26.548809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.548787 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq" Nov 03 20:27:26.557403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.557379 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:27:26.557717 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.557678 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:27:26.557717 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.557693 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:27:26.558560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.558511 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:27:26.559973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.559947 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq"] Nov 03 20:27:26.646366 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.646322 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9llzz\" (UniqueName: \"kubernetes.io/projected/1d7ffd96-9659-4080-b6bc-9343be470073-kube-api-access-9llzz\") pod \"maas-api-revoke-sub-47da3855-5gbwq\" (UID: \"1d7ffd96-9659-4080-b6bc-9343be470073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq" Nov 03 20:27:26.646627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.646469 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7ffd96-9659-4080-b6bc-9343be470073-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47da3855-5gbwq\" (UID: \"1d7ffd96-9659-4080-b6bc-9343be470073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq" Nov 03 20:27:26.646743 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.646723 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7ffd96-9659-4080-b6bc-9343be470073-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47da3855-5gbwq\" (UID: \"1d7ffd96-9659-4080-b6bc-9343be470073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq" Nov 03 20:27:26.747830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.747778 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7ffd96-9659-4080-b6bc-9343be470073-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47da3855-5gbwq\" (UID: \"1d7ffd96-9659-4080-b6bc-9343be470073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq" Nov 03 20:27:26.748017 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.747860 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9llzz\" (UniqueName: \"kubernetes.io/projected/1d7ffd96-9659-4080-b6bc-9343be470073-kube-api-access-9llzz\") pod \"maas-api-revoke-sub-47da3855-5gbwq\" (UID: \"1d7ffd96-9659-4080-b6bc-9343be470073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq" Nov 03 20:27:26.748017 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.747949 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7ffd96-9659-4080-b6bc-9343be470073-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47da3855-5gbwq\" (UID: \"1d7ffd96-9659-4080-b6bc-9343be470073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq" Nov 03 20:27:26.748646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.748620 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7ffd96-9659-4080-b6bc-9343be470073-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47da3855-5gbwq\" (UID: \"1d7ffd96-9659-4080-b6bc-9343be470073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq" Nov 03 20:27:26.750260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.750238 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7ffd96-9659-4080-b6bc-9343be470073-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47da3855-5gbwq\" (UID: \"1d7ffd96-9659-4080-b6bc-9343be470073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq" Nov 03 20:27:26.756236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.756213 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9llzz\" (UniqueName: \"kubernetes.io/projected/1d7ffd96-9659-4080-b6bc-9343be470073-kube-api-access-9llzz\") pod \"maas-api-revoke-sub-47da3855-5gbwq\" (UID: \"1d7ffd96-9659-4080-b6bc-9343be470073\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq" Nov 03 20:27:26.864836 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:26.864757 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq" Nov 03 20:27:27.142868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:27.142838 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq"] Nov 03 20:27:27.144238 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:27:27.144214 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7ffd96_9659_4080_b6bc_9343be470073.slice/crio-7674eb7d80d109a5e2dbc2ce02ea94d122b13598fb5dd20a518887914c2e6c31 WatchSource:0}: Error finding container 7674eb7d80d109a5e2dbc2ce02ea94d122b13598fb5dd20a518887914c2e6c31: Status 404 returned error can't find the container with id 7674eb7d80d109a5e2dbc2ce02ea94d122b13598fb5dd20a518887914c2e6c31 Nov 03 20:27:27.244188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:27.244155 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq" event={"ID":"1d7ffd96-9659-4080-b6bc-9343be470073","Type":"ContainerStarted","Data":"7674eb7d80d109a5e2dbc2ce02ea94d122b13598fb5dd20a518887914c2e6c31"} Nov 03 20:27:28.254091 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:28.254009 5111 generic.go:356] "Generic (PLEG): container finished" podID="1d7ffd96-9659-4080-b6bc-9343be470073" containerID="6df9e12e65d8ec659ccd9d21d74cff4d45501e8b74560469fcc47a7abfdb37bc" exitCode=0 Nov 03 20:27:28.254487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:28.254103 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq" event={"ID":"1d7ffd96-9659-4080-b6bc-9343be470073","Type":"ContainerDied","Data":"6df9e12e65d8ec659ccd9d21d74cff4d45501e8b74560469fcc47a7abfdb37bc"} Nov 03 20:27:29.412480 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:29.412453 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq" Nov 03 20:27:29.480405 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:29.480368 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d7ffd96-9659-4080-b6bc-9343be470073-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7ffd96-9659-4080-b6bc-9343be470073-maas-api-xks-ca\") pod \"1d7ffd96-9659-4080-b6bc-9343be470073\" (UID: \"1d7ffd96-9659-4080-b6bc-9343be470073\") " Nov 03 20:27:29.480630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:29.480570 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7ffd96-9659-4080-b6bc-9343be470073-kube-api-access-9llzz\" (UniqueName: \"kubernetes.io/projected/1d7ffd96-9659-4080-b6bc-9343be470073-kube-api-access-9llzz\") pod \"1d7ffd96-9659-4080-b6bc-9343be470073\" (UID: \"1d7ffd96-9659-4080-b6bc-9343be470073\") " Nov 03 20:27:29.480630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:29.480600 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d7ffd96-9659-4080-b6bc-9343be470073-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7ffd96-9659-4080-b6bc-9343be470073-maas-api-openshift-ca\") pod \"1d7ffd96-9659-4080-b6bc-9343be470073\" (UID: \"1d7ffd96-9659-4080-b6bc-9343be470073\") " Nov 03 20:27:29.481000 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:29.480971 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7ffd96-9659-4080-b6bc-9343be470073-maas-api-openshift-ca" pod "1d7ffd96-9659-4080-b6bc-9343be470073" (UID: "1d7ffd96-9659-4080-b6bc-9343be470073"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:27:29.482482 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:29.482453 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d7ffd96-9659-4080-b6bc-9343be470073-maas-api-xks-ca" pod "1d7ffd96-9659-4080-b6bc-9343be470073" (UID: "1d7ffd96-9659-4080-b6bc-9343be470073"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:27:29.482633 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:29.482617 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7ffd96-9659-4080-b6bc-9343be470073-kube-api-access-9llzz" pod "1d7ffd96-9659-4080-b6bc-9343be470073" (UID: "1d7ffd96-9659-4080-b6bc-9343be470073"). InnerVolumeSpecName "kube-api-access-9llzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:29.582733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:29.582644 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7ffd96-9659-4080-b6bc-9343be470073-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:29.582733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:29.582677 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9llzz\" (UniqueName: \"kubernetes.io/projected/1d7ffd96-9659-4080-b6bc-9343be470073-kube-api-access-9llzz\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:29.582733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:29.582688 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7ffd96-9659-4080-b6bc-9343be470073-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:30.270525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:30.270497 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq" Nov 03 20:27:30.270734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:30.270489 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq" event={"ID":"1d7ffd96-9659-4080-b6bc-9343be470073","Type":"ContainerDied","Data":"7674eb7d80d109a5e2dbc2ce02ea94d122b13598fb5dd20a518887914c2e6c31"} Nov 03 20:27:30.270734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:30.270629 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7674eb7d80d109a5e2dbc2ce02ea94d122b13598fb5dd20a518887914c2e6c31" Nov 03 20:27:35.056889 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:35.056846 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:27:35.644083 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:35.644012 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" containerID="cri-o://9323b22098eb0ae96f33e839bfe39ccb7b93b0b96508e029e2f612b91bec95cc" gracePeriod=45 Nov 03 20:27:35.657863 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:35.657827 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" containerID="cri-o://1fbc34268ceaa29c2fbf32e4e27da8be4c99d43e1d94bc361cd1808d60dad915" gracePeriod=45 Nov 03 20:27:35.666441 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:35.666409 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" containerID="cri-o://556af6cb6da2f941a88c53d85d342ad9d80966eb98c617038faa48ae2e393fc1" gracePeriod=45 Nov 03 20:27:35.975798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:35.975766 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:27:35.980759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:35.980729 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:27:35.983648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:35.983629 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:27:36.062193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062145 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-tmp-dir\") pod \"275b0913-a9c9-4165-b1b1-be6e85c7294a\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " Nov 03 20:27:36.062722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062202 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-home\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-home\") pod \"275b0913-a9c9-4165-b1b1-be6e85c7294a\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " Nov 03 20:27:36.062722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062247 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275b0913-a9c9-4165-b1b1-be6e85c7294a-kube-api-access-f87jf\" (UniqueName: \"kubernetes.io/projected/275b0913-a9c9-4165-b1b1-be6e85c7294a-kube-api-access-f87jf\") pod \"275b0913-a9c9-4165-b1b1-be6e85c7294a\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " Nov 03 20:27:36.062722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062310 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/275b0913-a9c9-4165-b1b1-be6e85c7294a-tls-certs\" (UniqueName: \"kubernetes.io/secret/275b0913-a9c9-4165-b1b1-be6e85c7294a-tls-certs\") pod \"275b0913-a9c9-4165-b1b1-be6e85c7294a\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " Nov 03 20:27:36.062722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062458 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-model-cache\") pod \"275b0913-a9c9-4165-b1b1-be6e85c7294a\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " Nov 03 20:27:36.062722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062501 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-model-cache\") pod \"4c4b3178-37d9-4185-ab51-19e7e7914725\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " Nov 03 20:27:36.062722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062564 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c4b3178-37d9-4185-ab51-19e7e7914725-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4b3178-37d9-4185-ab51-19e7e7914725-tls-certs\") pod \"4c4b3178-37d9-4185-ab51-19e7e7914725\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " Nov 03 20:27:36.062722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062600 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-tmp-dir" pod "275b0913-a9c9-4165-b1b1-be6e85c7294a" (UID: "275b0913-a9c9-4165-b1b1-be6e85c7294a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:36.062722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062618 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-home" pod "275b0913-a9c9-4165-b1b1-be6e85c7294a" (UID: "275b0913-a9c9-4165-b1b1-be6e85c7294a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:36.062722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062609 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-dshm\") pod \"4c4b3178-37d9-4185-ab51-19e7e7914725\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " Nov 03 20:27:36.062722 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062700 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-dshm\") pod \"275b0913-a9c9-4165-b1b1-be6e85c7294a\" (UID: \"275b0913-a9c9-4165-b1b1-be6e85c7294a\") " Nov 03 20:27:36.063269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062754 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-dshm\") pod \"ab2b92fb-74eb-42cc-a77a-26871be8a997\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " Nov 03 20:27:36.063269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062800 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-home\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-home\") pod \"4c4b3178-37d9-4185-ab51-19e7e7914725\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " Nov 03 20:27:36.063269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062857 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-model-cache" pod "4c4b3178-37d9-4185-ab51-19e7e7914725" (UID: "4c4b3178-37d9-4185-ab51-19e7e7914725"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:36.063269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062861 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-model-cache\") pod \"ab2b92fb-74eb-42cc-a77a-26871be8a997\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " Nov 03 20:27:36.063269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062917 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-tmp-dir\") pod \"ab2b92fb-74eb-42cc-a77a-26871be8a997\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " Nov 03 20:27:36.063269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.062988 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-tmp-dir\") pod \"4c4b3178-37d9-4185-ab51-19e7e7914725\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " Nov 03 20:27:36.063269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.063057 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab2b92fb-74eb-42cc-a77a-26871be8a997-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab2b92fb-74eb-42cc-a77a-26871be8a997-tls-certs\") pod \"ab2b92fb-74eb-42cc-a77a-26871be8a997\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " Nov 03 20:27:36.063269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.063061 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-model-cache" pod "ab2b92fb-74eb-42cc-a77a-26871be8a997" (UID: "ab2b92fb-74eb-42cc-a77a-26871be8a997"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:36.063269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.063088 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4b3178-37d9-4185-ab51-19e7e7914725-kube-api-access-gb5nl\" (UniqueName: \"kubernetes.io/projected/4c4b3178-37d9-4185-ab51-19e7e7914725-kube-api-access-gb5nl\") pod \"4c4b3178-37d9-4185-ab51-19e7e7914725\" (UID: \"4c4b3178-37d9-4185-ab51-19e7e7914725\") " Nov 03 20:27:36.063269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.063125 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-home\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-home\") pod \"ab2b92fb-74eb-42cc-a77a-26871be8a997\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " Nov 03 20:27:36.063269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.063160 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab2b92fb-74eb-42cc-a77a-26871be8a997-kube-api-access-8nm6c\" (UniqueName: \"kubernetes.io/projected/ab2b92fb-74eb-42cc-a77a-26871be8a997-kube-api-access-8nm6c\") pod \"ab2b92fb-74eb-42cc-a77a-26871be8a997\" (UID: \"ab2b92fb-74eb-42cc-a77a-26871be8a997\") " Nov 03 20:27:36.063875 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.063341 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-model-cache" pod "275b0913-a9c9-4165-b1b1-be6e85c7294a" (UID: "275b0913-a9c9-4165-b1b1-be6e85c7294a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:36.063875 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.063421 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-home" pod "4c4b3178-37d9-4185-ab51-19e7e7914725" (UID: "4c4b3178-37d9-4185-ab51-19e7e7914725"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:36.063875 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.063654 5111 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-model-cache\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.063875 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.063677 5111 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-model-cache\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.063875 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.063694 5111 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-home\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.063875 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.063709 5111 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-model-cache\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.063875 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.063723 5111 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-tmp-dir\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.063875 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.063736 5111 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-home\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.064314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.064004 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-home" pod "ab2b92fb-74eb-42cc-a77a-26871be8a997" (UID: "ab2b92fb-74eb-42cc-a77a-26871be8a997"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:36.064314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.064074 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-tmp-dir" pod "ab2b92fb-74eb-42cc-a77a-26871be8a997" (UID: "ab2b92fb-74eb-42cc-a77a-26871be8a997"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:36.064314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.064174 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-tmp-dir" pod "4c4b3178-37d9-4185-ab51-19e7e7914725" (UID: "4c4b3178-37d9-4185-ab51-19e7e7914725"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:36.067418 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.067359 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-dshm" pod "ab2b92fb-74eb-42cc-a77a-26871be8a997" (UID: "ab2b92fb-74eb-42cc-a77a-26871be8a997"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:36.067594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.067477 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-dshm" pod "4c4b3178-37d9-4185-ab51-19e7e7914725" (UID: "4c4b3178-37d9-4185-ab51-19e7e7914725"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:36.067594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.067477 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c4b3178-37d9-4185-ab51-19e7e7914725-tls-certs" pod "4c4b3178-37d9-4185-ab51-19e7e7914725" (UID: "4c4b3178-37d9-4185-ab51-19e7e7914725"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:27:36.067594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.067568 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab2b92fb-74eb-42cc-a77a-26871be8a997-kube-api-access-8nm6c" pod "ab2b92fb-74eb-42cc-a77a-26871be8a997" (UID: "ab2b92fb-74eb-42cc-a77a-26871be8a997"). InnerVolumeSpecName "kube-api-access-8nm6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:36.067793 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.067616 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275b0913-a9c9-4165-b1b1-be6e85c7294a-kube-api-access-f87jf" pod "275b0913-a9c9-4165-b1b1-be6e85c7294a" (UID: "275b0913-a9c9-4165-b1b1-be6e85c7294a"). InnerVolumeSpecName "kube-api-access-f87jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:36.068013 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.067990 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/275b0913-a9c9-4165-b1b1-be6e85c7294a-tls-certs" pod "275b0913-a9c9-4165-b1b1-be6e85c7294a" (UID: "275b0913-a9c9-4165-b1b1-be6e85c7294a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:27:36.068113 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.068077 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab2b92fb-74eb-42cc-a77a-26871be8a997-tls-certs" pod "ab2b92fb-74eb-42cc-a77a-26871be8a997" (UID: "ab2b92fb-74eb-42cc-a77a-26871be8a997"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:27:36.068673 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.068654 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4b3178-37d9-4185-ab51-19e7e7914725-kube-api-access-gb5nl" pod "4c4b3178-37d9-4185-ab51-19e7e7914725" (UID: "4c4b3178-37d9-4185-ab51-19e7e7914725"). InnerVolumeSpecName "kube-api-access-gb5nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:36.068763 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.068734 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-dshm" pod "275b0913-a9c9-4165-b1b1-be6e85c7294a" (UID: "275b0913-a9c9-4165-b1b1-be6e85c7294a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:36.164506 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.164456 5111 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-tmp-dir\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.164506 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.164497 5111 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-tmp-dir\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.164506 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.164510 5111 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab2b92fb-74eb-42cc-a77a-26871be8a997-tls-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.164824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.164522 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb5nl\" (UniqueName: \"kubernetes.io/projected/4c4b3178-37d9-4185-ab51-19e7e7914725-kube-api-access-gb5nl\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.164824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.164571 5111 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-home\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.164824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.164583 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nm6c\" (UniqueName: \"kubernetes.io/projected/ab2b92fb-74eb-42cc-a77a-26871be8a997-kube-api-access-8nm6c\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.164824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.164595 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f87jf\" (UniqueName: \"kubernetes.io/projected/275b0913-a9c9-4165-b1b1-be6e85c7294a-kube-api-access-f87jf\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.164824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.164607 5111 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275b0913-a9c9-4165-b1b1-be6e85c7294a-tls-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.164824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.164618 5111 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4b3178-37d9-4185-ab51-19e7e7914725-tls-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.164824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.164629 5111 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4b3178-37d9-4185-ab51-19e7e7914725-dshm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.164824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.164641 5111 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275b0913-a9c9-4165-b1b1-be6e85c7294a-dshm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.164824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.164651 5111 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab2b92fb-74eb-42cc-a77a-26871be8a997-dshm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:36.316180 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.316143 5111 generic.go:356] "Generic (PLEG): container finished" podID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerID="556af6cb6da2f941a88c53d85d342ad9d80966eb98c617038faa48ae2e393fc1" exitCode=0 Nov 03 20:27:36.316392 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.316187 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" Nov 03 20:27:36.316392 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.316224 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" event={"ID":"ab2b92fb-74eb-42cc-a77a-26871be8a997","Type":"ContainerDied","Data":"556af6cb6da2f941a88c53d85d342ad9d80966eb98c617038faa48ae2e393fc1"} Nov 03 20:27:36.316392 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.316269 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm" event={"ID":"ab2b92fb-74eb-42cc-a77a-26871be8a997","Type":"ContainerDied","Data":"3eab62860a4209d12b9653fd74145840c888abdd155e0101fb0fd5d85c78bd3e"} Nov 03 20:27:36.316392 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.316290 5111 scope.go:122] "RemoveContainer" containerID="556af6cb6da2f941a88c53d85d342ad9d80966eb98c617038faa48ae2e393fc1" Nov 03 20:27:36.317878 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.317824 5111 generic.go:356] "Generic (PLEG): container finished" podID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerID="9323b22098eb0ae96f33e839bfe39ccb7b93b0b96508e029e2f612b91bec95cc" exitCode=0 Nov 03 20:27:36.318009 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.317875 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" Nov 03 20:27:36.318009 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.317917 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" event={"ID":"4c4b3178-37d9-4185-ab51-19e7e7914725","Type":"ContainerDied","Data":"9323b22098eb0ae96f33e839bfe39ccb7b93b0b96508e029e2f612b91bec95cc"} Nov 03 20:27:36.318009 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.317966 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65" event={"ID":"4c4b3178-37d9-4185-ab51-19e7e7914725","Type":"ContainerDied","Data":"02f876c17622aca0cc146f2b120aa7df72519106008d5649cb0a59548ec85ba6"} Nov 03 20:27:36.319729 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.319705 5111 generic.go:356] "Generic (PLEG): container finished" podID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerID="1fbc34268ceaa29c2fbf32e4e27da8be4c99d43e1d94bc361cd1808d60dad915" exitCode=0 Nov 03 20:27:36.319858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.319744 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" Nov 03 20:27:36.319858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.319795 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" event={"ID":"275b0913-a9c9-4165-b1b1-be6e85c7294a","Type":"ContainerDied","Data":"1fbc34268ceaa29c2fbf32e4e27da8be4c99d43e1d94bc361cd1808d60dad915"} Nov 03 20:27:36.319858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.319830 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" event={"ID":"275b0913-a9c9-4165-b1b1-be6e85c7294a","Type":"ContainerDied","Data":"07513c9fe48f549e263a465a893fdf79fa44b4252d7a0336fffae2d394b7ec81"} Nov 03 20:27:36.328132 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.328113 5111 scope.go:122] "RemoveContainer" containerID="556af6cb6da2f941a88c53d85d342ad9d80966eb98c617038faa48ae2e393fc1" Nov 03 20:27:36.328421 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:36.328400 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556af6cb6da2f941a88c53d85d342ad9d80966eb98c617038faa48ae2e393fc1\": container with ID starting with 556af6cb6da2f941a88c53d85d342ad9d80966eb98c617038faa48ae2e393fc1 not found: ID does not exist" containerID="556af6cb6da2f941a88c53d85d342ad9d80966eb98c617038faa48ae2e393fc1" Nov 03 20:27:36.328470 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.328432 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556af6cb6da2f941a88c53d85d342ad9d80966eb98c617038faa48ae2e393fc1"} err="failed to get container status \"556af6cb6da2f941a88c53d85d342ad9d80966eb98c617038faa48ae2e393fc1\": rpc error: code = NotFound desc = could not find container \"556af6cb6da2f941a88c53d85d342ad9d80966eb98c617038faa48ae2e393fc1\": container with ID starting with 556af6cb6da2f941a88c53d85d342ad9d80966eb98c617038faa48ae2e393fc1 not found: ID does not exist" Nov 03 20:27:36.328470 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.328450 5111 scope.go:122] "RemoveContainer" containerID="9323b22098eb0ae96f33e839bfe39ccb7b93b0b96508e029e2f612b91bec95cc" Nov 03 20:27:36.335974 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.335949 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm"] Nov 03 20:27:36.338916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.338896 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/simulator-w2-11a5b0-kserve-66c6694b5f-5gxfm"] Nov 03 20:27:36.339679 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.339661 5111 scope.go:122] "RemoveContainer" containerID="9323b22098eb0ae96f33e839bfe39ccb7b93b0b96508e029e2f612b91bec95cc" Nov 03 20:27:36.340030 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:36.340006 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9323b22098eb0ae96f33e839bfe39ccb7b93b0b96508e029e2f612b91bec95cc\": container with ID starting with 9323b22098eb0ae96f33e839bfe39ccb7b93b0b96508e029e2f612b91bec95cc not found: ID does not exist" containerID="9323b22098eb0ae96f33e839bfe39ccb7b93b0b96508e029e2f612b91bec95cc" Nov 03 20:27:36.340114 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.340043 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9323b22098eb0ae96f33e839bfe39ccb7b93b0b96508e029e2f612b91bec95cc"} err="failed to get container status \"9323b22098eb0ae96f33e839bfe39ccb7b93b0b96508e029e2f612b91bec95cc\": rpc error: code = NotFound desc = could not find container \"9323b22098eb0ae96f33e839bfe39ccb7b93b0b96508e029e2f612b91bec95cc\": container with ID starting with 9323b22098eb0ae96f33e839bfe39ccb7b93b0b96508e029e2f612b91bec95cc not found: ID does not exist" Nov 03 20:27:36.340114 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.340067 5111 scope.go:122] "RemoveContainer" containerID="1fbc34268ceaa29c2fbf32e4e27da8be4c99d43e1d94bc361cd1808d60dad915" Nov 03 20:27:36.349855 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.349834 5111 scope.go:122] "RemoveContainer" containerID="1fbc34268ceaa29c2fbf32e4e27da8be4c99d43e1d94bc361cd1808d60dad915" Nov 03 20:27:36.350165 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:36.350103 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fbc34268ceaa29c2fbf32e4e27da8be4c99d43e1d94bc361cd1808d60dad915\": container with ID starting with 1fbc34268ceaa29c2fbf32e4e27da8be4c99d43e1d94bc361cd1808d60dad915 not found: ID does not exist" containerID="1fbc34268ceaa29c2fbf32e4e27da8be4c99d43e1d94bc361cd1808d60dad915" Nov 03 20:27:36.350233 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.350173 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbc34268ceaa29c2fbf32e4e27da8be4c99d43e1d94bc361cd1808d60dad915"} err="failed to get container status \"1fbc34268ceaa29c2fbf32e4e27da8be4c99d43e1d94bc361cd1808d60dad915\": rpc error: code = NotFound desc = could not find container \"1fbc34268ceaa29c2fbf32e4e27da8be4c99d43e1d94bc361cd1808d60dad915\": container with ID starting with 1fbc34268ceaa29c2fbf32e4e27da8be4c99d43e1d94bc361cd1808d60dad915 not found: ID does not exist" Nov 03 20:27:36.355280 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.355259 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks"] Nov 03 20:27:36.359260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.359237 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks"] Nov 03 20:27:36.369976 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.369946 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65"] Nov 03 20:27:36.373913 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.373891 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/distinct-w2-11a5b0-kserve-78f4d7b79c-rxm65"] Nov 03 20:27:36.923699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:36.923592 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/premium-w2-11a5b0-kserve-5ff7bc548b-fdcks" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:27:38.182128 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:38.182094 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" path="/var/lib/kubelet/pods/275b0913-a9c9-4165-b1b1-be6e85c7294a/volumes" Nov 03 20:27:38.182557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:38.182510 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" path="/var/lib/kubelet/pods/4c4b3178-37d9-4185-ab51-19e7e7914725/volumes" Nov 03 20:27:38.182990 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:38.182974 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" path="/var/lib/kubelet/pods/ab2b92fb-74eb-42cc-a77a-26871be8a997/volumes" Nov 03 20:27:41.962931 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:41.962885 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf"] Nov 03 20:27:41.965854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:41.965822 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-11a5b0-zjwqf"] Nov 03 20:27:42.181570 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:42.181506 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed18a08-f3d1-456c-a6a2-9b73f5eb866a" path="/var/lib/kubelet/pods/0ed18a08-f3d1-456c-a6a2-9b73f5eb866a/volumes" Nov 03 20:27:48.797005 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:48.796967 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s"] Nov 03 20:27:48.797466 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:48.797259 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" podUID="b205487d-c7cb-4491-a6e5-c90771872d3c" containerName="istio-proxy" containerID="cri-o://7337209cb74a6ef11f414a99069c2b9031a1b4440c15df98c39088b93ce9983d" gracePeriod=30 Nov 03 20:27:51.428853 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:51.428820 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f5dc4b79-ltr7j"] Nov 03 20:27:51.429413 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:51.429082 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" podUID="02f9d4b8-695a-4049-8bdb-1d62792ed657" containerName="manager" containerID="cri-o://6ac57524f9fcf7cfc4531cb12ef1045cb6cd7cb3d6cb6ef2ef5e15f297436d28" gracePeriod=30 Nov 03 20:27:51.688341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:51.688281 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" Nov 03 20:27:51.738445 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:51.738402 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f9d4b8-695a-4049-8bdb-1d62792ed657-kube-api-access-bfczw\" (UniqueName: \"kubernetes.io/projected/02f9d4b8-695a-4049-8bdb-1d62792ed657-kube-api-access-bfczw\") pod \"02f9d4b8-695a-4049-8bdb-1d62792ed657\" (UID: \"02f9d4b8-695a-4049-8bdb-1d62792ed657\") " Nov 03 20:27:51.738673 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:51.738492 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02f9d4b8-695a-4049-8bdb-1d62792ed657-webhook-cert\" (UniqueName: \"kubernetes.io/secret/02f9d4b8-695a-4049-8bdb-1d62792ed657-webhook-cert\") pod \"02f9d4b8-695a-4049-8bdb-1d62792ed657\" (UID: \"02f9d4b8-695a-4049-8bdb-1d62792ed657\") " Nov 03 20:27:51.738774 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:51.738687 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02f9d4b8-695a-4049-8bdb-1d62792ed657-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f9d4b8-695a-4049-8bdb-1d62792ed657-metrics-certs\") pod \"02f9d4b8-695a-4049-8bdb-1d62792ed657\" (UID: \"02f9d4b8-695a-4049-8bdb-1d62792ed657\") " Nov 03 20:27:51.741074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:51.741040 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f9d4b8-695a-4049-8bdb-1d62792ed657-metrics-certs" pod "02f9d4b8-695a-4049-8bdb-1d62792ed657" (UID: "02f9d4b8-695a-4049-8bdb-1d62792ed657"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:27:51.742282 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:51.742251 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f9d4b8-695a-4049-8bdb-1d62792ed657-kube-api-access-bfczw" pod "02f9d4b8-695a-4049-8bdb-1d62792ed657" (UID: "02f9d4b8-695a-4049-8bdb-1d62792ed657"). InnerVolumeSpecName "kube-api-access-bfczw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:51.742282 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:51.742258 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f9d4b8-695a-4049-8bdb-1d62792ed657-webhook-cert" pod "02f9d4b8-695a-4049-8bdb-1d62792ed657" (UID: "02f9d4b8-695a-4049-8bdb-1d62792ed657"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:27:51.839685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:51.839650 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfczw\" (UniqueName: \"kubernetes.io/projected/02f9d4b8-695a-4049-8bdb-1d62792ed657-kube-api-access-bfczw\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:51.839685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:51.839679 5111 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02f9d4b8-695a-4049-8bdb-1d62792ed657-webhook-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:51.839685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:51.839689 5111 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02f9d4b8-695a-4049-8bdb-1d62792ed657-metrics-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:52.439601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:52.439493 5111 generic.go:356] "Generic (PLEG): container finished" podID="02f9d4b8-695a-4049-8bdb-1d62792ed657" containerID="6ac57524f9fcf7cfc4531cb12ef1045cb6cd7cb3d6cb6ef2ef5e15f297436d28" exitCode=0 Nov 03 20:27:52.439601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:52.439560 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" Nov 03 20:27:52.439601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:52.439575 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" event={"ID":"02f9d4b8-695a-4049-8bdb-1d62792ed657","Type":"ContainerDied","Data":"6ac57524f9fcf7cfc4531cb12ef1045cb6cd7cb3d6cb6ef2ef5e15f297436d28"} Nov 03 20:27:52.440169 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:52.439632 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5dc4b79-ltr7j" event={"ID":"02f9d4b8-695a-4049-8bdb-1d62792ed657","Type":"ContainerDied","Data":"e62feb62c1c93b2fc88ba6fdbcd97f1cfc90f8c25a9b7b56bba77f848c545b49"} Nov 03 20:27:52.440169 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:52.439655 5111 scope.go:122] "RemoveContainer" containerID="6ac57524f9fcf7cfc4531cb12ef1045cb6cd7cb3d6cb6ef2ef5e15f297436d28" Nov 03 20:27:52.453767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:52.453747 5111 scope.go:122] "RemoveContainer" containerID="6ac57524f9fcf7cfc4531cb12ef1045cb6cd7cb3d6cb6ef2ef5e15f297436d28" Nov 03 20:27:52.454036 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:52.454019 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac57524f9fcf7cfc4531cb12ef1045cb6cd7cb3d6cb6ef2ef5e15f297436d28\": container with ID starting with 6ac57524f9fcf7cfc4531cb12ef1045cb6cd7cb3d6cb6ef2ef5e15f297436d28 not found: ID does not exist" containerID="6ac57524f9fcf7cfc4531cb12ef1045cb6cd7cb3d6cb6ef2ef5e15f297436d28" Nov 03 20:27:52.454101 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:52.454047 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac57524f9fcf7cfc4531cb12ef1045cb6cd7cb3d6cb6ef2ef5e15f297436d28"} err="failed to get container status \"6ac57524f9fcf7cfc4531cb12ef1045cb6cd7cb3d6cb6ef2ef5e15f297436d28\": rpc error: code = NotFound desc = could not find container \"6ac57524f9fcf7cfc4531cb12ef1045cb6cd7cb3d6cb6ef2ef5e15f297436d28\": container with ID starting with 6ac57524f9fcf7cfc4531cb12ef1045cb6cd7cb3d6cb6ef2ef5e15f297436d28 not found: ID does not exist" Nov 03 20:27:52.460611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:52.460587 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f5dc4b79-ltr7j"] Nov 03 20:27:52.464883 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:52.464860 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65f5dc4b79-ltr7j"] Nov 03 20:27:53.570154 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:53.570113 5111 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:27:53.570864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:53.570824 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" podUID="b205487d-c7cb-4491-a6e5-c90771872d3c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:27:54.176555 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.169595 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:27:54.191292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.191256 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f9d4b8-695a-4049-8bdb-1d62792ed657" path="/var/lib/kubelet/pods/02f9d4b8-695a-4049-8bdb-1d62792ed657/volumes" Nov 03 20:27:54.322981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.322938 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-data\") pod \"b205487d-c7cb-4491-a6e5-c90771872d3c\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " Nov 03 20:27:54.322981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.322996 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-workload-certs\") pod \"b205487d-c7cb-4491-a6e5-c90771872d3c\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " Nov 03 20:27:54.323285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.323025 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b205487d-c7cb-4491-a6e5-c90771872d3c-kube-api-access-jzkbx\" (UniqueName: \"kubernetes.io/projected/b205487d-c7cb-4491-a6e5-c90771872d3c-kube-api-access-jzkbx\") pod \"b205487d-c7cb-4491-a6e5-c90771872d3c\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " Nov 03 20:27:54.323285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.323176 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-podinfo\") pod \"b205487d-c7cb-4491-a6e5-c90771872d3c\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " Nov 03 20:27:54.323285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.323217 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-token\" (UniqueName: \"kubernetes.io/projected/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-token\") pod \"b205487d-c7cb-4491-a6e5-c90771872d3c\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " Nov 03 20:27:54.323285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.323268 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-credential-socket\") pod \"b205487d-c7cb-4491-a6e5-c90771872d3c\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " Nov 03 20:27:54.323509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.323303 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-workload-socket\") pod \"b205487d-c7cb-4491-a6e5-c90771872d3c\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " Nov 03 20:27:54.323509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.323337 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-data" pod "b205487d-c7cb-4491-a6e5-c90771872d3c" (UID: "b205487d-c7cb-4491-a6e5-c90771872d3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:54.323509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.323353 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-workload-certs" pod "b205487d-c7cb-4491-a6e5-c90771872d3c" (UID: "b205487d-c7cb-4491-a6e5-c90771872d3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:54.323509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.323396 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-envoy\") pod \"b205487d-c7cb-4491-a6e5-c90771872d3c\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " Nov 03 20:27:54.323509 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.323435 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b205487d-c7cb-4491-a6e5-c90771872d3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b205487d-c7cb-4491-a6e5-c90771872d3c-istiod-ca-cert\") pod \"b205487d-c7cb-4491-a6e5-c90771872d3c\" (UID: \"b205487d-c7cb-4491-a6e5-c90771872d3c\") " Nov 03 20:27:54.323783 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.323657 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-workload-socket" pod "b205487d-c7cb-4491-a6e5-c90771872d3c" (UID: "b205487d-c7cb-4491-a6e5-c90771872d3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:54.323783 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.323657 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-credential-socket" pod "b205487d-c7cb-4491-a6e5-c90771872d3c" (UID: "b205487d-c7cb-4491-a6e5-c90771872d3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:54.323949 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.323927 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:54.324082 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.324065 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:54.324201 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.324186 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:54.324316 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.324304 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:54.324436 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.323995 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b205487d-c7cb-4491-a6e5-c90771872d3c-istiod-ca-cert" pod "b205487d-c7cb-4491-a6e5-c90771872d3c" (UID: "b205487d-c7cb-4491-a6e5-c90771872d3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:27:54.325664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.325627 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-podinfo" pod "b205487d-c7cb-4491-a6e5-c90771872d3c" (UID: "b205487d-c7cb-4491-a6e5-c90771872d3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:27:54.325846 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.325825 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b205487d-c7cb-4491-a6e5-c90771872d3c-kube-api-access-jzkbx" pod "b205487d-c7cb-4491-a6e5-c90771872d3c" (UID: "b205487d-c7cb-4491-a6e5-c90771872d3c"). InnerVolumeSpecName "kube-api-access-jzkbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:54.325994 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.325977 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-token" pod "b205487d-c7cb-4491-a6e5-c90771872d3c" (UID: "b205487d-c7cb-4491-a6e5-c90771872d3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:54.326041 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.326000 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-envoy" pod "b205487d-c7cb-4491-a6e5-c90771872d3c" (UID: "b205487d-c7cb-4491-a6e5-c90771872d3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:54.425034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.424937 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:54.425034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.424981 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b205487d-c7cb-4491-a6e5-c90771872d3c-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:54.425034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.424995 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzkbx\" (UniqueName: \"kubernetes.io/projected/b205487d-c7cb-4491-a6e5-c90771872d3c-kube-api-access-jzkbx\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:54.425034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.425006 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:54.425034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.425014 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b205487d-c7cb-4491-a6e5-c90771872d3c-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:54.463297 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.463253 5111 generic.go:356] "Generic (PLEG): container finished" podID="b205487d-c7cb-4491-a6e5-c90771872d3c" containerID="7337209cb74a6ef11f414a99069c2b9031a1b4440c15df98c39088b93ce9983d" exitCode=0 Nov 03 20:27:54.463786 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.463754 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" event={"ID":"b205487d-c7cb-4491-a6e5-c90771872d3c","Type":"ContainerDied","Data":"7337209cb74a6ef11f414a99069c2b9031a1b4440c15df98c39088b93ce9983d"} Nov 03 20:27:54.463999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.463980 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" event={"ID":"b205487d-c7cb-4491-a6e5-c90771872d3c","Type":"ContainerDied","Data":"9f22b791d73ab540bd7cd9575b39fe3585efc1d3e92679674acf7eb184127e91"} Nov 03 20:27:54.464142 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.464126 5111 scope.go:122] "RemoveContainer" containerID="7337209cb74a6ef11f414a99069c2b9031a1b4440c15df98c39088b93ce9983d" Nov 03 20:27:54.464376 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.464358 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s" Nov 03 20:27:54.523166 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.523134 5111 scope.go:122] "RemoveContainer" containerID="7337209cb74a6ef11f414a99069c2b9031a1b4440c15df98c39088b93ce9983d" Nov 03 20:27:54.523512 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:27:54.523484 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7337209cb74a6ef11f414a99069c2b9031a1b4440c15df98c39088b93ce9983d\": container with ID starting with 7337209cb74a6ef11f414a99069c2b9031a1b4440c15df98c39088b93ce9983d not found: ID does not exist" containerID="7337209cb74a6ef11f414a99069c2b9031a1b4440c15df98c39088b93ce9983d" Nov 03 20:27:54.523648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.523522 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7337209cb74a6ef11f414a99069c2b9031a1b4440c15df98c39088b93ce9983d"} err="failed to get container status \"7337209cb74a6ef11f414a99069c2b9031a1b4440c15df98c39088b93ce9983d\": rpc error: code = NotFound desc = could not find container \"7337209cb74a6ef11f414a99069c2b9031a1b4440c15df98c39088b93ce9983d\": container with ID starting with 7337209cb74a6ef11f414a99069c2b9031a1b4440c15df98c39088b93ce9983d not found: ID does not exist" Nov 03 20:27:54.529558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.529512 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s"] Nov 03 20:27:54.537094 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:54.537064 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-2bn5s"] Nov 03 20:27:56.181754 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:27:56.181719 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b205487d-c7cb-4491-a6e5-c90771872d3c" path="/var/lib/kubelet/pods/b205487d-c7cb-4491-a6e5-c90771872d3c/volumes" Nov 03 20:28:05.056938 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:28:05.056899 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:28:14.122768 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.122680 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8"] Nov 03 20:28:14.123197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123180 5111 cpu_manager.go:414] "containerMap: removing container" podUID="b205487d-c7cb-4491-a6e5-c90771872d3c" containerName="istio-proxy" Nov 03 20:28:14.123240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123199 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b205487d-c7cb-4491-a6e5-c90771872d3c" containerName="istio-proxy" Nov 03 20:28:14.123240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123214 5111 cpu_manager.go:414] "containerMap: removing container" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" Nov 03 20:28:14.123240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123220 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" Nov 03 20:28:14.123240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123229 5111 cpu_manager.go:414] "containerMap: removing container" podUID="1d7ffd96-9659-4080-b6bc-9343be470073" containerName="revoke-keys" Nov 03 20:28:14.123240 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123234 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7ffd96-9659-4080-b6bc-9343be470073" containerName="revoke-keys" Nov 03 20:28:14.123407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123245 5111 cpu_manager.go:414] "containerMap: removing container" podUID="02f9d4b8-695a-4049-8bdb-1d62792ed657" containerName="manager" Nov 03 20:28:14.123407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123249 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f9d4b8-695a-4049-8bdb-1d62792ed657" containerName="manager" Nov 03 20:28:14.123407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123255 5111 cpu_manager.go:414] "containerMap: removing container" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" Nov 03 20:28:14.123407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123259 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" Nov 03 20:28:14.123407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123273 5111 cpu_manager.go:414] "containerMap: removing container" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerName="istio-proxy" Nov 03 20:28:14.123407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123277 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2815d45f-e02c-4178-9f95-0375fda5f11c" containerName="istio-proxy" Nov 03 20:28:14.123407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123286 5111 cpu_manager.go:414] "containerMap: removing container" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" Nov 03 20:28:14.123407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123290 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" Nov 03 20:28:14.123407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123374 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="b205487d-c7cb-4491-a6e5-c90771872d3c" containerName="istio-proxy" Nov 03 20:28:14.123407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123381 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="275b0913-a9c9-4165-b1b1-be6e85c7294a" containerName="main" Nov 03 20:28:14.123407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123389 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7ffd96-9659-4080-b6bc-9343be470073" containerName="revoke-keys" Nov 03 20:28:14.123407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123395 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f9d4b8-695a-4049-8bdb-1d62792ed657" containerName="manager" Nov 03 20:28:14.123407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123401 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4b3178-37d9-4185-ab51-19e7e7914725" containerName="main" Nov 03 20:28:14.123407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.123408 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2b92fb-74eb-42cc-a77a-26871be8a997" containerName="main" Nov 03 20:28:14.126772 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.126755 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8" Nov 03 20:28:14.132333 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.132307 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:28:14.132523 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.132486 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:28:14.133960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.133930 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8"] Nov 03 20:28:14.144372 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.144331 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:28:14.144523 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.144379 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:28:14.229198 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.229158 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8jht8\" (UID: \"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8" Nov 03 20:28:14.229397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.229217 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46xmr\" (UniqueName: \"kubernetes.io/projected/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-kube-api-access-46xmr\") pod \"maas-api-revoke-sub-85237e45-8jht8\" (UID: \"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8" Nov 03 20:28:14.229444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.229396 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8jht8\" (UID: \"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8" Nov 03 20:28:14.330656 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.330602 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8jht8\" (UID: \"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8" Nov 03 20:28:14.330842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.330690 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8jht8\" (UID: \"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8" Nov 03 20:28:14.330842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.330723 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46xmr\" (UniqueName: \"kubernetes.io/projected/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-kube-api-access-46xmr\") pod \"maas-api-revoke-sub-85237e45-8jht8\" (UID: \"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8" Nov 03 20:28:14.331464 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.331444 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8jht8\" (UID: \"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8" Nov 03 20:28:14.333176 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.333156 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8jht8\" (UID: \"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8" Nov 03 20:28:14.338656 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.338606 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46xmr\" (UniqueName: \"kubernetes.io/projected/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-kube-api-access-46xmr\") pod \"maas-api-revoke-sub-85237e45-8jht8\" (UID: \"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8" Nov 03 20:28:14.445110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.445024 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8" Nov 03 20:28:14.725936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:14.725908 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8"] Nov 03 20:28:14.727805 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:28:14.727769 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c3dda0_fde6_4c15_bf6f_323f4c0a7ccd.slice/crio-bd4555bded3e8809de5b3520fce44b293a118a91c257d8f199183b75ebc08d23 WatchSource:0}: Error finding container bd4555bded3e8809de5b3520fce44b293a118a91c257d8f199183b75ebc08d23: Status 404 returned error can't find the container with id bd4555bded3e8809de5b3520fce44b293a118a91c257d8f199183b75ebc08d23 Nov 03 20:28:15.607890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:15.607854 5111 generic.go:356] "Generic (PLEG): container finished" podID="a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd" containerID="6bd878f23050ffe12b78ca473c2c6c4fef6c5c55ec8ba7baa2f5f06d401eaff1" exitCode=0 Nov 03 20:28:15.608283 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:15.607932 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8" event={"ID":"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd","Type":"ContainerDied","Data":"6bd878f23050ffe12b78ca473c2c6c4fef6c5c55ec8ba7baa2f5f06d401eaff1"} Nov 03 20:28:15.608283 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:15.607989 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8" event={"ID":"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd","Type":"ContainerStarted","Data":"bd4555bded3e8809de5b3520fce44b293a118a91c257d8f199183b75ebc08d23"} Nov 03 20:28:16.762482 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:16.762452 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8" Nov 03 20:28:16.858400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:16.858356 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-maas-api-xks-ca\") pod \"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd\" (UID: \"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd\") " Nov 03 20:28:16.858673 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:16.858430 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-maas-api-openshift-ca\") pod \"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd\" (UID: \"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd\") " Nov 03 20:28:16.858673 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:16.858595 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-kube-api-access-46xmr\" (UniqueName: \"kubernetes.io/projected/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-kube-api-access-46xmr\") pod \"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd\" (UID: \"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd\") " Nov 03 20:28:16.858928 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:16.858897 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-maas-api-openshift-ca" pod "a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd" (UID: "a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:28:16.859013 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:16.858994 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:16.860695 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:16.860663 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-maas-api-xks-ca" pod "a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd" (UID: "a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:28:16.860817 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:16.860750 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-kube-api-access-46xmr" pod "a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd" (UID: "a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd"). InnerVolumeSpecName "kube-api-access-46xmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:16.960507 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:16.960470 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46xmr\" (UniqueName: \"kubernetes.io/projected/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-kube-api-access-46xmr\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:16.960507 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:16.960501 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:17.624068 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:17.624031 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8" event={"ID":"a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd","Type":"ContainerDied","Data":"bd4555bded3e8809de5b3520fce44b293a118a91c257d8f199183b75ebc08d23"} Nov 03 20:28:17.624238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:17.624085 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd4555bded3e8809de5b3520fce44b293a118a91c257d8f199183b75ebc08d23" Nov 03 20:28:17.624238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:17.624133 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8" Nov 03 20:28:35.056664 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:28:35.056626 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:28:59.023467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:59.023397 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc"] Nov 03 20:28:59.026744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:59.026709 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2"] Nov 03 20:28:59.029163 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:59.029150 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f7ff14-n5hmc"] Nov 03 20:28:59.032302 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:28:59.032275 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-364c1787-c2tm2"] Nov 03 20:29:00.181832 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:00.181798 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ebbac93-2f53-40cb-904f-d8a25729e213" path="/var/lib/kubelet/pods/1ebbac93-2f53-40cb-904f-d8a25729e213/volumes" Nov 03 20:29:00.182249 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:00.182224 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe4e11b-d14b-4b23-83d4-6815212acbed" path="/var/lib/kubelet/pods/fbe4e11b-d14b-4b23-83d4-6815212acbed/volumes" Nov 03 20:29:05.080762 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:29:05.080717 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:29:09.578412 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.578377 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g"] Nov 03 20:29:09.578965 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.578873 5111 cpu_manager.go:414] "containerMap: removing container" podUID="a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd" containerName="revoke-keys" Nov 03 20:29:09.578965 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.578890 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd" containerName="revoke-keys" Nov 03 20:29:09.579056 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.578997 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd" containerName="revoke-keys" Nov 03 20:29:09.582386 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.582369 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g" Nov 03 20:29:09.587206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.586908 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:29:09.587206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.586931 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:29:09.587206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.587065 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:29:09.587206 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.587140 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:29:09.589444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.589415 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g"] Nov 03 20:29:09.700343 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.700304 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb44g\" (UniqueName: \"kubernetes.io/projected/9a27dec6-8bae-4276-8255-4acdb766880b-kube-api-access-zb44g\") pod \"maas-api-revoke-sub-c1e271a9-vls8g\" (UID: \"9a27dec6-8bae-4276-8255-4acdb766880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g" Nov 03 20:29:09.700561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.700387 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a27dec6-8bae-4276-8255-4acdb766880b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vls8g\" (UID: \"9a27dec6-8bae-4276-8255-4acdb766880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g" Nov 03 20:29:09.700561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.700510 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a27dec6-8bae-4276-8255-4acdb766880b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vls8g\" (UID: \"9a27dec6-8bae-4276-8255-4acdb766880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g" Nov 03 20:29:09.801365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.801325 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a27dec6-8bae-4276-8255-4acdb766880b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vls8g\" (UID: \"9a27dec6-8bae-4276-8255-4acdb766880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g" Nov 03 20:29:09.801640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.801439 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a27dec6-8bae-4276-8255-4acdb766880b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vls8g\" (UID: \"9a27dec6-8bae-4276-8255-4acdb766880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g" Nov 03 20:29:09.801640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.801503 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb44g\" (UniqueName: \"kubernetes.io/projected/9a27dec6-8bae-4276-8255-4acdb766880b-kube-api-access-zb44g\") pod \"maas-api-revoke-sub-c1e271a9-vls8g\" (UID: \"9a27dec6-8bae-4276-8255-4acdb766880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g" Nov 03 20:29:09.802324 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.802289 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a27dec6-8bae-4276-8255-4acdb766880b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vls8g\" (UID: \"9a27dec6-8bae-4276-8255-4acdb766880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g" Nov 03 20:29:09.803930 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.803900 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a27dec6-8bae-4276-8255-4acdb766880b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vls8g\" (UID: \"9a27dec6-8bae-4276-8255-4acdb766880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g" Nov 03 20:29:09.809419 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.809397 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb44g\" (UniqueName: \"kubernetes.io/projected/9a27dec6-8bae-4276-8255-4acdb766880b-kube-api-access-zb44g\") pod \"maas-api-revoke-sub-c1e271a9-vls8g\" (UID: \"9a27dec6-8bae-4276-8255-4acdb766880b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g" Nov 03 20:29:09.900187 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:09.900153 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g" Nov 03 20:29:10.179048 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:29:10.179015 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a27dec6_8bae_4276_8255_4acdb766880b.slice/crio-03d4ebcb0328b917fb72cbd4b157911b63abc17d30bb446e126326423760253e WatchSource:0}: Error finding container 03d4ebcb0328b917fb72cbd4b157911b63abc17d30bb446e126326423760253e: Status 404 returned error can't find the container with id 03d4ebcb0328b917fb72cbd4b157911b63abc17d30bb446e126326423760253e Nov 03 20:29:10.182773 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:10.182745 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g"] Nov 03 20:29:10.990654 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:10.990626 5111 generic.go:356] "Generic (PLEG): container finished" podID="9a27dec6-8bae-4276-8255-4acdb766880b" containerID="08c2bc39a68ba143752657c97262589f3f59d114a857d9779d05464af1781f4f" exitCode=0 Nov 03 20:29:10.991029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:10.990705 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g" event={"ID":"9a27dec6-8bae-4276-8255-4acdb766880b","Type":"ContainerDied","Data":"08c2bc39a68ba143752657c97262589f3f59d114a857d9779d05464af1781f4f"} Nov 03 20:29:10.991029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:10.990762 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g" event={"ID":"9a27dec6-8bae-4276-8255-4acdb766880b","Type":"ContainerStarted","Data":"03d4ebcb0328b917fb72cbd4b157911b63abc17d30bb446e126326423760253e"} Nov 03 20:29:12.141833 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:12.141806 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g" Nov 03 20:29:12.228743 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:12.228700 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a27dec6-8bae-4276-8255-4acdb766880b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a27dec6-8bae-4276-8255-4acdb766880b-maas-api-xks-ca\") pod \"9a27dec6-8bae-4276-8255-4acdb766880b\" (UID: \"9a27dec6-8bae-4276-8255-4acdb766880b\") " Nov 03 20:29:12.228937 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:12.228801 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a27dec6-8bae-4276-8255-4acdb766880b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a27dec6-8bae-4276-8255-4acdb766880b-maas-api-openshift-ca\") pod \"9a27dec6-8bae-4276-8255-4acdb766880b\" (UID: \"9a27dec6-8bae-4276-8255-4acdb766880b\") " Nov 03 20:29:12.228937 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:12.228864 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a27dec6-8bae-4276-8255-4acdb766880b-kube-api-access-zb44g\" (UniqueName: \"kubernetes.io/projected/9a27dec6-8bae-4276-8255-4acdb766880b-kube-api-access-zb44g\") pod \"9a27dec6-8bae-4276-8255-4acdb766880b\" (UID: \"9a27dec6-8bae-4276-8255-4acdb766880b\") " Nov 03 20:29:12.229194 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:12.229151 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a27dec6-8bae-4276-8255-4acdb766880b-maas-api-openshift-ca" pod "9a27dec6-8bae-4276-8255-4acdb766880b" (UID: "9a27dec6-8bae-4276-8255-4acdb766880b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:12.230982 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:12.230962 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a27dec6-8bae-4276-8255-4acdb766880b-maas-api-xks-ca" pod "9a27dec6-8bae-4276-8255-4acdb766880b" (UID: "9a27dec6-8bae-4276-8255-4acdb766880b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:29:12.231122 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:12.231101 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a27dec6-8bae-4276-8255-4acdb766880b-kube-api-access-zb44g" pod "9a27dec6-8bae-4276-8255-4acdb766880b" (UID: "9a27dec6-8bae-4276-8255-4acdb766880b"). InnerVolumeSpecName "kube-api-access-zb44g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:12.330422 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:12.330333 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a27dec6-8bae-4276-8255-4acdb766880b-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.330422 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:12.330367 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a27dec6-8bae-4276-8255-4acdb766880b-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.330422 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:12.330380 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb44g\" (UniqueName: \"kubernetes.io/projected/9a27dec6-8bae-4276-8255-4acdb766880b-kube-api-access-zb44g\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:13.005884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:13.005846 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g" Nov 03 20:29:13.006084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:13.005883 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g" event={"ID":"9a27dec6-8bae-4276-8255-4acdb766880b","Type":"ContainerDied","Data":"03d4ebcb0328b917fb72cbd4b157911b63abc17d30bb446e126326423760253e"} Nov 03 20:29:13.006084 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:13.005973 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03d4ebcb0328b917fb72cbd4b157911b63abc17d30bb446e126326423760253e" Nov 03 20:29:28.243681 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.243643 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4"] Nov 03 20:29:28.244243 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.244221 5111 cpu_manager.go:414] "containerMap: removing container" podUID="9a27dec6-8bae-4276-8255-4acdb766880b" containerName="revoke-keys" Nov 03 20:29:28.244290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.244246 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a27dec6-8bae-4276-8255-4acdb766880b" containerName="revoke-keys" Nov 03 20:29:28.244355 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.244345 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a27dec6-8bae-4276-8255-4acdb766880b" containerName="revoke-keys" Nov 03 20:29:28.247921 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.247895 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4" Nov 03 20:29:28.252153 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.251907 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:29:28.253384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.253357 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:29:28.253605 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.253412 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:29:28.253767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.253584 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:29:28.257815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.257786 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4"] Nov 03 20:29:28.356868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.356827 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d61b606-c574-4963-b13e-669efa986122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ghcn4\" (UID: \"6d61b606-c574-4963-b13e-669efa986122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4" Nov 03 20:29:28.357052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.356875 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d61b606-c574-4963-b13e-669efa986122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ghcn4\" (UID: \"6d61b606-c574-4963-b13e-669efa986122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4" Nov 03 20:29:28.357052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.357005 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dqsk\" (UniqueName: \"kubernetes.io/projected/6d61b606-c574-4963-b13e-669efa986122-kube-api-access-6dqsk\") pod \"maas-api-revoke-sub-2e1f5b5f-ghcn4\" (UID: \"6d61b606-c574-4963-b13e-669efa986122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4" Nov 03 20:29:28.458591 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.458527 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d61b606-c574-4963-b13e-669efa986122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ghcn4\" (UID: \"6d61b606-c574-4963-b13e-669efa986122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4" Nov 03 20:29:28.458795 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.458601 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d61b606-c574-4963-b13e-669efa986122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ghcn4\" (UID: \"6d61b606-c574-4963-b13e-669efa986122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4" Nov 03 20:29:28.458795 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.458638 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dqsk\" (UniqueName: \"kubernetes.io/projected/6d61b606-c574-4963-b13e-669efa986122-kube-api-access-6dqsk\") pod \"maas-api-revoke-sub-2e1f5b5f-ghcn4\" (UID: \"6d61b606-c574-4963-b13e-669efa986122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4" Nov 03 20:29:28.459283 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.459255 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d61b606-c574-4963-b13e-669efa986122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ghcn4\" (UID: \"6d61b606-c574-4963-b13e-669efa986122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4" Nov 03 20:29:28.460933 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.460910 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d61b606-c574-4963-b13e-669efa986122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ghcn4\" (UID: \"6d61b606-c574-4963-b13e-669efa986122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4" Nov 03 20:29:28.467362 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.467333 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dqsk\" (UniqueName: \"kubernetes.io/projected/6d61b606-c574-4963-b13e-669efa986122-kube-api-access-6dqsk\") pod \"maas-api-revoke-sub-2e1f5b5f-ghcn4\" (UID: \"6d61b606-c574-4963-b13e-669efa986122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4" Nov 03 20:29:28.564170 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.564083 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4" Nov 03 20:29:28.849354 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:28.849324 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4"] Nov 03 20:29:28.850787 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:29:28.850750 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d61b606_c574_4963_b13e_669efa986122.slice/crio-ab68855be1646b6b99cfb3643591ca27c510f1a95330424f3820a40302adcfbe WatchSource:0}: Error finding container ab68855be1646b6b99cfb3643591ca27c510f1a95330424f3820a40302adcfbe: Status 404 returned error can't find the container with id ab68855be1646b6b99cfb3643591ca27c510f1a95330424f3820a40302adcfbe Nov 03 20:29:29.163485 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:29.163457 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4" event={"ID":"6d61b606-c574-4963-b13e-669efa986122","Type":"ContainerStarted","Data":"ab68855be1646b6b99cfb3643591ca27c510f1a95330424f3820a40302adcfbe"} Nov 03 20:29:30.180610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:30.180571 5111 generic.go:356] "Generic (PLEG): container finished" podID="6d61b606-c574-4963-b13e-669efa986122" containerID="80b8a5d840778d12ce7b19f11f8366d62beec8ee018f54f575dab8471868d7f3" exitCode=0 Nov 03 20:29:30.184271 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:30.184236 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4" event={"ID":"6d61b606-c574-4963-b13e-669efa986122","Type":"ContainerDied","Data":"80b8a5d840778d12ce7b19f11f8366d62beec8ee018f54f575dab8471868d7f3"} Nov 03 20:29:30.936214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:30.936183 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:29:30.939706 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:30.939679 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:29:30.941374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:30.941355 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:29:30.944973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:30.944951 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:29:31.329203 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:31.329176 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4" Nov 03 20:29:31.389105 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:31.389069 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d61b606-c574-4963-b13e-669efa986122-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d61b606-c574-4963-b13e-669efa986122-maas-api-xks-ca\") pod \"6d61b606-c574-4963-b13e-669efa986122\" (UID: \"6d61b606-c574-4963-b13e-669efa986122\") " Nov 03 20:29:31.389278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:31.389122 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d61b606-c574-4963-b13e-669efa986122-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d61b606-c574-4963-b13e-669efa986122-maas-api-openshift-ca\") pod \"6d61b606-c574-4963-b13e-669efa986122\" (UID: \"6d61b606-c574-4963-b13e-669efa986122\") " Nov 03 20:29:31.389278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:31.389169 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d61b606-c574-4963-b13e-669efa986122-kube-api-access-6dqsk\" (UniqueName: \"kubernetes.io/projected/6d61b606-c574-4963-b13e-669efa986122-kube-api-access-6dqsk\") pod \"6d61b606-c574-4963-b13e-669efa986122\" (UID: \"6d61b606-c574-4963-b13e-669efa986122\") " Nov 03 20:29:31.389582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:31.389551 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d61b606-c574-4963-b13e-669efa986122-maas-api-openshift-ca" pod "6d61b606-c574-4963-b13e-669efa986122" (UID: "6d61b606-c574-4963-b13e-669efa986122"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:31.391290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:31.391266 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d61b606-c574-4963-b13e-669efa986122-maas-api-xks-ca" pod "6d61b606-c574-4963-b13e-669efa986122" (UID: "6d61b606-c574-4963-b13e-669efa986122"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:29:31.391290 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:31.391272 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d61b606-c574-4963-b13e-669efa986122-kube-api-access-6dqsk" pod "6d61b606-c574-4963-b13e-669efa986122" (UID: "6d61b606-c574-4963-b13e-669efa986122"). InnerVolumeSpecName "kube-api-access-6dqsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:31.490194 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:31.490100 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d61b606-c574-4963-b13e-669efa986122-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:31.490194 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:31.490132 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d61b606-c574-4963-b13e-669efa986122-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:31.490194 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:31.490144 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dqsk\" (UniqueName: \"kubernetes.io/projected/6d61b606-c574-4963-b13e-669efa986122-kube-api-access-6dqsk\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:32.194279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:32.194249 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4" event={"ID":"6d61b606-c574-4963-b13e-669efa986122","Type":"ContainerDied","Data":"ab68855be1646b6b99cfb3643591ca27c510f1a95330424f3820a40302adcfbe"} Nov 03 20:29:32.194448 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:32.194279 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4" Nov 03 20:29:32.194448 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:32.194297 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab68855be1646b6b99cfb3643591ca27c510f1a95330424f3820a40302adcfbe" Nov 03 20:29:35.057670 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:29:35.057629 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:29:38.362488 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:38.362453 5111 scope.go:122] "RemoveContainer" containerID="eb75b9b471554f04ab33a3465e345ebc4f9e9c982989167066b30cb11dab67d2" Nov 03 20:29:38.372315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:38.372290 5111 scope.go:122] "RemoveContainer" containerID="242f4e9e3f8f8e34edb84ac368f94923bb8a5e0841b69043bc882911042f34db" Nov 03 20:29:39.231356 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.231323 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d"] Nov 03 20:29:39.231837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.231822 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6d61b606-c574-4963-b13e-669efa986122" containerName="revoke-keys" Nov 03 20:29:39.231895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.231840 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d61b606-c574-4963-b13e-669efa986122" containerName="revoke-keys" Nov 03 20:29:39.231940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.231922 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d61b606-c574-4963-b13e-669efa986122" containerName="revoke-keys" Nov 03 20:29:39.236815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.236783 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" Nov 03 20:29:39.241433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.241404 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:29:39.241629 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.241565 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:29:39.241629 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.241627 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:29:39.241935 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.241916 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:29:39.243746 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.243718 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d"] Nov 03 20:29:39.374766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.374723 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc43082d-8921-4f7a-b1a1-5e2336501117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-k9h5d\" (UID: \"fc43082d-8921-4f7a-b1a1-5e2336501117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" Nov 03 20:29:39.374766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.374771 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc43082d-8921-4f7a-b1a1-5e2336501117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-k9h5d\" (UID: \"fc43082d-8921-4f7a-b1a1-5e2336501117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" Nov 03 20:29:39.375350 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.374871 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6mpk\" (UniqueName: \"kubernetes.io/projected/fc43082d-8921-4f7a-b1a1-5e2336501117-kube-api-access-s6mpk\") pod \"maas-api-revoke-sub-ec843018-k9h5d\" (UID: \"fc43082d-8921-4f7a-b1a1-5e2336501117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" Nov 03 20:29:39.475849 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.475806 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc43082d-8921-4f7a-b1a1-5e2336501117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-k9h5d\" (UID: \"fc43082d-8921-4f7a-b1a1-5e2336501117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" Nov 03 20:29:39.476068 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.475869 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc43082d-8921-4f7a-b1a1-5e2336501117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-k9h5d\" (UID: \"fc43082d-8921-4f7a-b1a1-5e2336501117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" Nov 03 20:29:39.476068 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.475941 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6mpk\" (UniqueName: \"kubernetes.io/projected/fc43082d-8921-4f7a-b1a1-5e2336501117-kube-api-access-s6mpk\") pod \"maas-api-revoke-sub-ec843018-k9h5d\" (UID: \"fc43082d-8921-4f7a-b1a1-5e2336501117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" Nov 03 20:29:39.477134 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.477100 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc43082d-8921-4f7a-b1a1-5e2336501117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-k9h5d\" (UID: \"fc43082d-8921-4f7a-b1a1-5e2336501117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" Nov 03 20:29:39.479955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.479920 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc43082d-8921-4f7a-b1a1-5e2336501117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-k9h5d\" (UID: \"fc43082d-8921-4f7a-b1a1-5e2336501117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" Nov 03 20:29:39.484263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.484206 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6mpk\" (UniqueName: \"kubernetes.io/projected/fc43082d-8921-4f7a-b1a1-5e2336501117-kube-api-access-s6mpk\") pod \"maas-api-revoke-sub-ec843018-k9h5d\" (UID: \"fc43082d-8921-4f7a-b1a1-5e2336501117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" Nov 03 20:29:39.553569 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.553520 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" Nov 03 20:29:39.840805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:39.840772 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d"] Nov 03 20:29:39.841564 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:29:39.841508 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc43082d_8921_4f7a_b1a1_5e2336501117.slice/crio-f7a82dac8f7075c28cad352b5f260ee8c765a184b29554906fae23daf0ab9949 WatchSource:0}: Error finding container f7a82dac8f7075c28cad352b5f260ee8c765a184b29554906fae23daf0ab9949: Status 404 returned error can't find the container with id f7a82dac8f7075c28cad352b5f260ee8c765a184b29554906fae23daf0ab9949 Nov 03 20:29:40.253384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:40.253338 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" event={"ID":"fc43082d-8921-4f7a-b1a1-5e2336501117","Type":"ContainerStarted","Data":"9875c99d590bfc5ed230d80d065d2cddfed6709f1bd15940d0607ab5ff3d203d"} Nov 03 20:29:40.253620 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:40.253426 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" event={"ID":"fc43082d-8921-4f7a-b1a1-5e2336501117","Type":"ContainerStarted","Data":"f7a82dac8f7075c28cad352b5f260ee8c765a184b29554906fae23daf0ab9949"} Nov 03 20:29:40.273040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:40.272985 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" podStartSLOduration=1.27297049 podStartE2EDuration="1.27297049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:40.269022816 +0000 UTC m=+3010.231401255" watchObservedRunningTime="2026-11-03 20:29:40.27297049 +0000 UTC m=+3010.235348951" Nov 03 20:29:41.264643 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:41.264599 5111 generic.go:356] "Generic (PLEG): container finished" podID="fc43082d-8921-4f7a-b1a1-5e2336501117" containerID="9875c99d590bfc5ed230d80d065d2cddfed6709f1bd15940d0607ab5ff3d203d" exitCode=0 Nov 03 20:29:41.265277 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:41.264693 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" event={"ID":"fc43082d-8921-4f7a-b1a1-5e2336501117","Type":"ContainerDied","Data":"9875c99d590bfc5ed230d80d065d2cddfed6709f1bd15940d0607ab5ff3d203d"} Nov 03 20:29:42.761759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:42.761725 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" Nov 03 20:29:42.829048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:42.827353 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc43082d-8921-4f7a-b1a1-5e2336501117-kube-api-access-s6mpk\" (UniqueName: \"kubernetes.io/projected/fc43082d-8921-4f7a-b1a1-5e2336501117-kube-api-access-s6mpk\") pod \"fc43082d-8921-4f7a-b1a1-5e2336501117\" (UID: \"fc43082d-8921-4f7a-b1a1-5e2336501117\") " Nov 03 20:29:42.829048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:42.827450 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc43082d-8921-4f7a-b1a1-5e2336501117-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc43082d-8921-4f7a-b1a1-5e2336501117-maas-api-openshift-ca\") pod \"fc43082d-8921-4f7a-b1a1-5e2336501117\" (UID: \"fc43082d-8921-4f7a-b1a1-5e2336501117\") " Nov 03 20:29:42.829048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:42.827488 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc43082d-8921-4f7a-b1a1-5e2336501117-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc43082d-8921-4f7a-b1a1-5e2336501117-maas-api-xks-ca\") pod \"fc43082d-8921-4f7a-b1a1-5e2336501117\" (UID: \"fc43082d-8921-4f7a-b1a1-5e2336501117\") " Nov 03 20:29:42.829048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:42.828987 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc43082d-8921-4f7a-b1a1-5e2336501117-maas-api-openshift-ca" pod "fc43082d-8921-4f7a-b1a1-5e2336501117" (UID: "fc43082d-8921-4f7a-b1a1-5e2336501117"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:42.832043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:42.832011 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc43082d-8921-4f7a-b1a1-5e2336501117-kube-api-access-s6mpk" pod "fc43082d-8921-4f7a-b1a1-5e2336501117" (UID: "fc43082d-8921-4f7a-b1a1-5e2336501117"). InnerVolumeSpecName "kube-api-access-s6mpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:42.833792 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:42.833770 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc43082d-8921-4f7a-b1a1-5e2336501117-maas-api-xks-ca" pod "fc43082d-8921-4f7a-b1a1-5e2336501117" (UID: "fc43082d-8921-4f7a-b1a1-5e2336501117"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:29:42.928472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:42.928431 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6mpk\" (UniqueName: \"kubernetes.io/projected/fc43082d-8921-4f7a-b1a1-5e2336501117-kube-api-access-s6mpk\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:42.928472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:42.928458 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc43082d-8921-4f7a-b1a1-5e2336501117-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:42.928472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:42.928469 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc43082d-8921-4f7a-b1a1-5e2336501117-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:43.323841 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:43.323790 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" Nov 03 20:29:43.324055 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:43.323802 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d" event={"ID":"fc43082d-8921-4f7a-b1a1-5e2336501117","Type":"ContainerDied","Data":"f7a82dac8f7075c28cad352b5f260ee8c765a184b29554906fae23daf0ab9949"} Nov 03 20:29:43.324055 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:43.323907 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7a82dac8f7075c28cad352b5f260ee8c765a184b29554906fae23daf0ab9949" Nov 03 20:29:57.021662 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:57.021625 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8"] Nov 03 20:29:57.024434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:57.024411 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c4b5a4-kjrr8"] Nov 03 20:29:58.183086 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:29:58.183053 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2" path="/var/lib/kubelet/pods/6ff3ad5e-85a6-4f86-944c-bdeb4197d1c2/volumes" Nov 03 20:30:00.139068 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.139034 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl"] Nov 03 20:30:00.139541 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.139516 5111 cpu_manager.go:414] "containerMap: removing container" podUID="fc43082d-8921-4f7a-b1a1-5e2336501117" containerName="revoke-keys" Nov 03 20:30:00.139599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.139560 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc43082d-8921-4f7a-b1a1-5e2336501117" containerName="revoke-keys" Nov 03 20:30:00.139654 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.139648 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc43082d-8921-4f7a-b1a1-5e2336501117" containerName="revoke-keys" Nov 03 20:30:00.144072 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.144050 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" Nov 03 20:30:00.149007 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.148972 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:30:00.149159 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.149012 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:30:00.149159 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.149027 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:30:00.150627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.150598 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl"] Nov 03 20:30:00.220765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.220726 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/057e26f5-ad1c-4f69-9983-bb6b4bf09aac-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895630-nztzl\" (UID: \"057e26f5-ad1c-4f69-9983-bb6b4bf09aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" Nov 03 20:30:00.220979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.220955 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9csqz\" (UniqueName: \"kubernetes.io/projected/057e26f5-ad1c-4f69-9983-bb6b4bf09aac-kube-api-access-9csqz\") pod \"maas-api-key-cleanup-29895630-nztzl\" (UID: \"057e26f5-ad1c-4f69-9983-bb6b4bf09aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" Nov 03 20:30:00.321975 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.321931 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9csqz\" (UniqueName: \"kubernetes.io/projected/057e26f5-ad1c-4f69-9983-bb6b4bf09aac-kube-api-access-9csqz\") pod \"maas-api-key-cleanup-29895630-nztzl\" (UID: \"057e26f5-ad1c-4f69-9983-bb6b4bf09aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" Nov 03 20:30:00.322188 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.321989 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/057e26f5-ad1c-4f69-9983-bb6b4bf09aac-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895630-nztzl\" (UID: \"057e26f5-ad1c-4f69-9983-bb6b4bf09aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" Nov 03 20:30:00.322738 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.322715 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/057e26f5-ad1c-4f69-9983-bb6b4bf09aac-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895630-nztzl\" (UID: \"057e26f5-ad1c-4f69-9983-bb6b4bf09aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" Nov 03 20:30:00.330394 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.330357 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9csqz\" (UniqueName: \"kubernetes.io/projected/057e26f5-ad1c-4f69-9983-bb6b4bf09aac-kube-api-access-9csqz\") pod \"maas-api-key-cleanup-29895630-nztzl\" (UID: \"057e26f5-ad1c-4f69-9983-bb6b4bf09aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" Nov 03 20:30:00.459887 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.459807 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" Nov 03 20:30:00.757907 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:00.757869 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl"] Nov 03 20:30:01.499423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:01.499383 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" event={"ID":"057e26f5-ad1c-4f69-9983-bb6b4bf09aac","Type":"ContainerStarted","Data":"c4f2f1b75a2a55589825c69333502cff0da77789d1f2a07ea05c0bbdc847df25"} Nov 03 20:30:01.499931 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:01.499435 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" event={"ID":"057e26f5-ad1c-4f69-9983-bb6b4bf09aac","Type":"ContainerStarted","Data":"5f4e22f49bdeaf8bba12c17f88811e1465167ac1cd2fa1e1c140dd67992c97f0"} Nov 03 20:30:01.520549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:01.520484 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" podStartSLOduration=1.5204694220000001 podStartE2EDuration="1.520469422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:01.518633968 +0000 UTC m=+3031.481012406" watchObservedRunningTime="2026-11-03 20:30:01.520469422 +0000 UTC m=+3031.482847852" Nov 03 20:30:02.508061 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:02.508026 5111 generic.go:356] "Generic (PLEG): container finished" podID="057e26f5-ad1c-4f69-9983-bb6b4bf09aac" containerID="c4f2f1b75a2a55589825c69333502cff0da77789d1f2a07ea05c0bbdc847df25" exitCode=60 Nov 03 20:30:02.508561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:02.508113 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" event={"ID":"057e26f5-ad1c-4f69-9983-bb6b4bf09aac","Type":"ContainerDied","Data":"c4f2f1b75a2a55589825c69333502cff0da77789d1f2a07ea05c0bbdc847df25"} Nov 03 20:30:02.508561 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:02.508379 5111 scope.go:122] "RemoveContainer" containerID="c4f2f1b75a2a55589825c69333502cff0da77789d1f2a07ea05c0bbdc847df25" Nov 03 20:30:03.529276 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:03.529240 5111 generic.go:356] "Generic (PLEG): container finished" podID="057e26f5-ad1c-4f69-9983-bb6b4bf09aac" containerID="4c3a19e943bc8fe93013cf95faddbe1eb67a1d4fd847668f30ec01829a69aded" exitCode=60 Nov 03 20:30:03.529778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:03.529330 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" event={"ID":"057e26f5-ad1c-4f69-9983-bb6b4bf09aac","Type":"ContainerDied","Data":"4c3a19e943bc8fe93013cf95faddbe1eb67a1d4fd847668f30ec01829a69aded"} Nov 03 20:30:03.529778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:03.529424 5111 scope.go:122] "RemoveContainer" containerID="c4f2f1b75a2a55589825c69333502cff0da77789d1f2a07ea05c0bbdc847df25" Nov 03 20:30:03.529778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:03.529663 5111 scope.go:122] "RemoveContainer" containerID="4c3a19e943bc8fe93013cf95faddbe1eb67a1d4fd847668f30ec01829a69aded" Nov 03 20:30:03.529975 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:30:03.529947 5111 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895630-nztzl_odh-ai-gateway-infra(057e26f5-ad1c-4f69-9983-bb6b4bf09aac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" podUID="057e26f5-ad1c-4f69-9983-bb6b4bf09aac" Nov 03 20:30:04.539519 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:04.539491 5111 scope.go:122] "RemoveContainer" containerID="4c3a19e943bc8fe93013cf95faddbe1eb67a1d4fd847668f30ec01829a69aded" Nov 03 20:30:04.539941 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:30:04.539778 5111 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895630-nztzl_odh-ai-gateway-infra(057e26f5-ad1c-4f69-9983-bb6b4bf09aac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" podUID="057e26f5-ad1c-4f69-9983-bb6b4bf09aac" Nov 03 20:30:05.057641 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:30:05.057602 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:30:10.022717 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:10.022683 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866"] Nov 03 20:30:10.029568 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:10.029517 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1f511b2-2r866"] Nov 03 20:30:10.181664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:10.181625 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da3f92a8-fbf6-48e0-bf03-6d4e907e1421" path="/var/lib/kubelet/pods/da3f92a8-fbf6-48e0-bf03-6d4e907e1421/volumes" Nov 03 20:30:11.022969 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:11.022931 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c"] Nov 03 20:30:11.025487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:11.025465 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f563427-cgc4c"] Nov 03 20:30:12.182266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:12.182233 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1fc2582-7554-4e8d-ae03-a6864f3a15c9" path="/var/lib/kubelet/pods/d1fc2582-7554-4e8d-ae03-a6864f3a15c9/volumes" Nov 03 20:30:14.024072 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:14.024039 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59"] Nov 03 20:30:14.027121 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:14.027097 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac3af0e-c5k59"] Nov 03 20:30:14.182143 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:14.182112 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0954b165-7182-489a-b508-c21b4b21b0a9" path="/var/lib/kubelet/pods/0954b165-7182-489a-b508-c21b4b21b0a9/volumes" Nov 03 20:30:17.176097 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:17.176050 5111 scope.go:122] "RemoveContainer" containerID="4c3a19e943bc8fe93013cf95faddbe1eb67a1d4fd847668f30ec01829a69aded" Nov 03 20:30:17.634298 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:17.634255 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" event={"ID":"057e26f5-ad1c-4f69-9983-bb6b4bf09aac","Type":"ContainerStarted","Data":"4b86719d0500910886b8d57954d5d12d17b31a93e9dd30ba7569b94c50c2fb30"} Nov 03 20:30:18.642498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:18.642461 5111 generic.go:356] "Generic (PLEG): container finished" podID="057e26f5-ad1c-4f69-9983-bb6b4bf09aac" containerID="4b86719d0500910886b8d57954d5d12d17b31a93e9dd30ba7569b94c50c2fb30" exitCode=60 Nov 03 20:30:18.643022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:18.642547 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" event={"ID":"057e26f5-ad1c-4f69-9983-bb6b4bf09aac","Type":"ContainerDied","Data":"4b86719d0500910886b8d57954d5d12d17b31a93e9dd30ba7569b94c50c2fb30"} Nov 03 20:30:18.643022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:18.642636 5111 scope.go:122] "RemoveContainer" containerID="4c3a19e943bc8fe93013cf95faddbe1eb67a1d4fd847668f30ec01829a69aded" Nov 03 20:30:18.643022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:18.642982 5111 scope.go:122] "RemoveContainer" containerID="4b86719d0500910886b8d57954d5d12d17b31a93e9dd30ba7569b94c50c2fb30" Nov 03 20:30:18.643264 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:30:18.643243 5111 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895630-nztzl_odh-ai-gateway-infra(057e26f5-ad1c-4f69-9983-bb6b4bf09aac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" podUID="057e26f5-ad1c-4f69-9983-bb6b4bf09aac" Nov 03 20:30:18.681624 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:18.681590 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl"] Nov 03 20:30:19.799114 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:19.799089 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" Nov 03 20:30:19.841407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:19.841369 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/057e26f5-ad1c-4f69-9983-bb6b4bf09aac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/057e26f5-ad1c-4f69-9983-bb6b4bf09aac-maas-api-service-ca\") pod \"057e26f5-ad1c-4f69-9983-bb6b4bf09aac\" (UID: \"057e26f5-ad1c-4f69-9983-bb6b4bf09aac\") " Nov 03 20:30:19.841868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:19.841838 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057e26f5-ad1c-4f69-9983-bb6b4bf09aac-maas-api-service-ca" pod "057e26f5-ad1c-4f69-9983-bb6b4bf09aac" (UID: "057e26f5-ad1c-4f69-9983-bb6b4bf09aac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:19.942135 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:19.942048 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057e26f5-ad1c-4f69-9983-bb6b4bf09aac-kube-api-access-9csqz\" (UniqueName: \"kubernetes.io/projected/057e26f5-ad1c-4f69-9983-bb6b4bf09aac-kube-api-access-9csqz\") pod \"057e26f5-ad1c-4f69-9983-bb6b4bf09aac\" (UID: \"057e26f5-ad1c-4f69-9983-bb6b4bf09aac\") " Nov 03 20:30:19.942300 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:19.942262 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/057e26f5-ad1c-4f69-9983-bb6b4bf09aac-maas-api-service-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:19.944125 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:19.944101 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057e26f5-ad1c-4f69-9983-bb6b4bf09aac-kube-api-access-9csqz" pod "057e26f5-ad1c-4f69-9983-bb6b4bf09aac" (UID: "057e26f5-ad1c-4f69-9983-bb6b4bf09aac"). InnerVolumeSpecName "kube-api-access-9csqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.042914 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:20.042878 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9csqz\" (UniqueName: \"kubernetes.io/projected/057e26f5-ad1c-4f69-9983-bb6b4bf09aac-kube-api-access-9csqz\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.659242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:20.659211 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" Nov 03 20:30:20.659464 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:20.659249 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl" event={"ID":"057e26f5-ad1c-4f69-9983-bb6b4bf09aac","Type":"ContainerDied","Data":"5f4e22f49bdeaf8bba12c17f88811e1465167ac1cd2fa1e1c140dd67992c97f0"} Nov 03 20:30:20.659464 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:20.659311 5111 scope.go:122] "RemoveContainer" containerID="4b86719d0500910886b8d57954d5d12d17b31a93e9dd30ba7569b94c50c2fb30" Nov 03 20:30:20.679821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:20.679795 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl"] Nov 03 20:30:20.686011 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:20.685981 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-nztzl"] Nov 03 20:30:21.023731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:21.023644 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7"] Nov 03 20:30:21.028054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:21.028027 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a625455e-vrdt7"] Nov 03 20:30:22.181972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:22.181939 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="057e26f5-ad1c-4f69-9983-bb6b4bf09aac" path="/var/lib/kubelet/pods/057e26f5-ad1c-4f69-9983-bb6b4bf09aac/volumes" Nov 03 20:30:22.182418 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:22.182404 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d1e9568-9815-46be-bfff-746ed30973e1" path="/var/lib/kubelet/pods/3d1e9568-9815-46be-bfff-746ed30973e1/volumes" Nov 03 20:30:23.025248 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:23.025211 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8"] Nov 03 20:30:23.027571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:23.027547 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5d8dc4-ljpm8"] Nov 03 20:30:24.181079 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:24.181045 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b9f28c-e165-436f-a5a5-ae89d046297f" path="/var/lib/kubelet/pods/95b9f28c-e165-436f-a5a5-ae89d046297f/volumes" Nov 03 20:30:33.021979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:33.021940 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4"] Nov 03 20:30:33.025608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:33.025583 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fed3e4e-spfv4"] Nov 03 20:30:34.180932 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:34.180899 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db6cac85-00c7-479b-9de7-52c66c2749aa" path="/var/lib/kubelet/pods/db6cac85-00c7-479b-9de7-52c66c2749aa/volumes" Nov 03 20:30:35.066511 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:30:35.066443 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:30:36.347449 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.347407 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm"] Nov 03 20:30:36.347907 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.347892 5111 cpu_manager.go:414] "containerMap: removing container" podUID="057e26f5-ad1c-4f69-9983-bb6b4bf09aac" containerName="cleanup" Nov 03 20:30:36.347955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.347910 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057e26f5-ad1c-4f69-9983-bb6b4bf09aac" containerName="cleanup" Nov 03 20:30:36.347955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.347941 5111 cpu_manager.go:414] "containerMap: removing container" podUID="057e26f5-ad1c-4f69-9983-bb6b4bf09aac" containerName="cleanup" Nov 03 20:30:36.347955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.347948 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057e26f5-ad1c-4f69-9983-bb6b4bf09aac" containerName="cleanup" Nov 03 20:30:36.348062 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.348043 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="057e26f5-ad1c-4f69-9983-bb6b4bf09aac" containerName="cleanup" Nov 03 20:30:36.348062 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.348053 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="057e26f5-ad1c-4f69-9983-bb6b4bf09aac" containerName="cleanup" Nov 03 20:30:36.351609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.351592 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm" Nov 03 20:30:36.356026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.356001 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:30:36.356026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.356009 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:30:36.356794 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.356775 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:30:36.356941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.356893 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:30:36.359353 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.359328 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm"] Nov 03 20:30:36.475619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.475575 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bba6ee8-1f32-43bc-9da6-345fb14a9011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p5dwm\" (UID: \"1bba6ee8-1f32-43bc-9da6-345fb14a9011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm" Nov 03 20:30:36.475619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.475617 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bba6ee8-1f32-43bc-9da6-345fb14a9011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p5dwm\" (UID: \"1bba6ee8-1f32-43bc-9da6-345fb14a9011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm" Nov 03 20:30:36.475867 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.475635 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4kjh\" (UniqueName: \"kubernetes.io/projected/1bba6ee8-1f32-43bc-9da6-345fb14a9011-kube-api-access-v4kjh\") pod \"maas-api-revoke-sub-ae594979-p5dwm\" (UID: \"1bba6ee8-1f32-43bc-9da6-345fb14a9011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm" Nov 03 20:30:36.576358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.576323 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bba6ee8-1f32-43bc-9da6-345fb14a9011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p5dwm\" (UID: \"1bba6ee8-1f32-43bc-9da6-345fb14a9011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm" Nov 03 20:30:36.576358 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.576362 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bba6ee8-1f32-43bc-9da6-345fb14a9011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p5dwm\" (UID: \"1bba6ee8-1f32-43bc-9da6-345fb14a9011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm" Nov 03 20:30:36.576639 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.576382 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4kjh\" (UniqueName: \"kubernetes.io/projected/1bba6ee8-1f32-43bc-9da6-345fb14a9011-kube-api-access-v4kjh\") pod \"maas-api-revoke-sub-ae594979-p5dwm\" (UID: \"1bba6ee8-1f32-43bc-9da6-345fb14a9011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm" Nov 03 20:30:36.577027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.577005 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bba6ee8-1f32-43bc-9da6-345fb14a9011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-p5dwm\" (UID: \"1bba6ee8-1f32-43bc-9da6-345fb14a9011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm" Nov 03 20:30:36.578895 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.578874 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bba6ee8-1f32-43bc-9da6-345fb14a9011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-p5dwm\" (UID: \"1bba6ee8-1f32-43bc-9da6-345fb14a9011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm" Nov 03 20:30:36.584002 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.583977 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4kjh\" (UniqueName: \"kubernetes.io/projected/1bba6ee8-1f32-43bc-9da6-345fb14a9011-kube-api-access-v4kjh\") pod \"maas-api-revoke-sub-ae594979-p5dwm\" (UID: \"1bba6ee8-1f32-43bc-9da6-345fb14a9011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm" Nov 03 20:30:36.667949 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.667913 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm" Nov 03 20:30:36.943491 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:36.943455 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm"] Nov 03 20:30:36.945943 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:30:36.945915 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bba6ee8_1f32_43bc_9da6_345fb14a9011.slice/crio-b66e8795fe61f058824bca65be8277da6596cd9bf6b33b9a7c2972dc1aa44cee WatchSource:0}: Error finding container b66e8795fe61f058824bca65be8277da6596cd9bf6b33b9a7c2972dc1aa44cee: Status 404 returned error can't find the container with id b66e8795fe61f058824bca65be8277da6596cd9bf6b33b9a7c2972dc1aa44cee Nov 03 20:30:37.843958 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:37.843921 5111 generic.go:356] "Generic (PLEG): container finished" podID="1bba6ee8-1f32-43bc-9da6-345fb14a9011" containerID="f911a6fb1404e50715c182d6345b4117e77bc2c6bf1ce1a60b70c5175668dd60" exitCode=0 Nov 03 20:30:37.844422 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:37.844008 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm" event={"ID":"1bba6ee8-1f32-43bc-9da6-345fb14a9011","Type":"ContainerDied","Data":"f911a6fb1404e50715c182d6345b4117e77bc2c6bf1ce1a60b70c5175668dd60"} Nov 03 20:30:37.844422 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:37.844096 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm" event={"ID":"1bba6ee8-1f32-43bc-9da6-345fb14a9011","Type":"ContainerStarted","Data":"b66e8795fe61f058824bca65be8277da6596cd9bf6b33b9a7c2972dc1aa44cee"} Nov 03 20:30:38.407982 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:38.407945 5111 scope.go:122] "RemoveContainer" containerID="1ffd3cd0bb7b3127c81162845e22453434dc64ed5f5934ee29e9932ac3f194c2" Nov 03 20:30:38.417728 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:38.417702 5111 scope.go:122] "RemoveContainer" containerID="4fa27356325fad63671b69b771198284207c6d41ab4e2a20ccb38ed21d3e2a9a" Nov 03 20:30:38.427027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:38.427006 5111 scope.go:122] "RemoveContainer" containerID="e9fe4cb7404c6381223617692836d1ea11e1d435220e365b629aa9e48927e0a9" Nov 03 20:30:39.003380 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:39.003347 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm" Nov 03 20:30:39.097434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:39.097402 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bba6ee8-1f32-43bc-9da6-345fb14a9011-kube-api-access-v4kjh\" (UniqueName: \"kubernetes.io/projected/1bba6ee8-1f32-43bc-9da6-345fb14a9011-kube-api-access-v4kjh\") pod \"1bba6ee8-1f32-43bc-9da6-345fb14a9011\" (UID: \"1bba6ee8-1f32-43bc-9da6-345fb14a9011\") " Nov 03 20:30:39.097644 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:39.097453 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bba6ee8-1f32-43bc-9da6-345fb14a9011-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bba6ee8-1f32-43bc-9da6-345fb14a9011-maas-api-xks-ca\") pod \"1bba6ee8-1f32-43bc-9da6-345fb14a9011\" (UID: \"1bba6ee8-1f32-43bc-9da6-345fb14a9011\") " Nov 03 20:30:39.097644 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:39.097602 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bba6ee8-1f32-43bc-9da6-345fb14a9011-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bba6ee8-1f32-43bc-9da6-345fb14a9011-maas-api-openshift-ca\") pod \"1bba6ee8-1f32-43bc-9da6-345fb14a9011\" (UID: \"1bba6ee8-1f32-43bc-9da6-345fb14a9011\") " Nov 03 20:30:39.097973 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:39.097950 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bba6ee8-1f32-43bc-9da6-345fb14a9011-maas-api-openshift-ca" pod "1bba6ee8-1f32-43bc-9da6-345fb14a9011" (UID: "1bba6ee8-1f32-43bc-9da6-345fb14a9011"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:39.099612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:39.099589 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bba6ee8-1f32-43bc-9da6-345fb14a9011-maas-api-xks-ca" pod "1bba6ee8-1f32-43bc-9da6-345fb14a9011" (UID: "1bba6ee8-1f32-43bc-9da6-345fb14a9011"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:39.099740 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:39.099726 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bba6ee8-1f32-43bc-9da6-345fb14a9011-kube-api-access-v4kjh" pod "1bba6ee8-1f32-43bc-9da6-345fb14a9011" (UID: "1bba6ee8-1f32-43bc-9da6-345fb14a9011"). InnerVolumeSpecName "kube-api-access-v4kjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:39.199123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:39.199091 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4kjh\" (UniqueName: \"kubernetes.io/projected/1bba6ee8-1f32-43bc-9da6-345fb14a9011-kube-api-access-v4kjh\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:39.199123 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:39.199122 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bba6ee8-1f32-43bc-9da6-345fb14a9011-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:39.199327 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:39.199135 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bba6ee8-1f32-43bc-9da6-345fb14a9011-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:39.859803 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:39.859774 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm" Nov 03 20:30:39.859997 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:39.859814 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm" event={"ID":"1bba6ee8-1f32-43bc-9da6-345fb14a9011","Type":"ContainerDied","Data":"b66e8795fe61f058824bca65be8277da6596cd9bf6b33b9a7c2972dc1aa44cee"} Nov 03 20:30:39.859997 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:39.859859 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b66e8795fe61f058824bca65be8277da6596cd9bf6b33b9a7c2972dc1aa44cee" Nov 03 20:30:50.021836 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:50.021785 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb"] Nov 03 20:30:50.024642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:50.024612 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a9eb5a-hnsgb"] Nov 03 20:30:50.221744 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:50.221713 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2" path="/var/lib/kubelet/pods/52854a5d-5b2b-4b2f-85d5-889b7b0e6eb2/volumes" Nov 03 20:30:54.577201 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.577164 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh"] Nov 03 20:30:54.577671 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.577656 5111 cpu_manager.go:414] "containerMap: removing container" podUID="1bba6ee8-1f32-43bc-9da6-345fb14a9011" containerName="revoke-keys" Nov 03 20:30:54.577721 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.577672 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bba6ee8-1f32-43bc-9da6-345fb14a9011" containerName="revoke-keys" Nov 03 20:30:54.577721 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.577705 5111 cpu_manager.go:414] "containerMap: removing container" podUID="057e26f5-ad1c-4f69-9983-bb6b4bf09aac" containerName="cleanup" Nov 03 20:30:54.577721 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.577711 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057e26f5-ad1c-4f69-9983-bb6b4bf09aac" containerName="cleanup" Nov 03 20:30:54.577830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.577784 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bba6ee8-1f32-43bc-9da6-345fb14a9011" containerName="revoke-keys" Nov 03 20:30:54.577830 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.577794 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="057e26f5-ad1c-4f69-9983-bb6b4bf09aac" containerName="cleanup" Nov 03 20:30:54.582187 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.582168 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh" Nov 03 20:30:54.586284 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.586260 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:30:54.586474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.586263 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:30:54.586474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.586301 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:30:54.587231 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.587209 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:30:54.588501 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.588477 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh"] Nov 03 20:30:54.674666 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.674623 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a930808-b419-4475-8c2e-c13d888236f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6hwjh\" (UID: \"9a930808-b419-4475-8c2e-c13d888236f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh" Nov 03 20:30:54.674666 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.674667 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a930808-b419-4475-8c2e-c13d888236f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6hwjh\" (UID: \"9a930808-b419-4475-8c2e-c13d888236f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh" Nov 03 20:30:54.674923 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.674793 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqsz7\" (UniqueName: \"kubernetes.io/projected/9a930808-b419-4475-8c2e-c13d888236f6-kube-api-access-jqsz7\") pod \"maas-api-revoke-sub-a4aef7d7-6hwjh\" (UID: \"9a930808-b419-4475-8c2e-c13d888236f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh" Nov 03 20:30:54.775325 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.775281 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqsz7\" (UniqueName: \"kubernetes.io/projected/9a930808-b419-4475-8c2e-c13d888236f6-kube-api-access-jqsz7\") pod \"maas-api-revoke-sub-a4aef7d7-6hwjh\" (UID: \"9a930808-b419-4475-8c2e-c13d888236f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh" Nov 03 20:30:54.775574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.775429 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a930808-b419-4475-8c2e-c13d888236f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6hwjh\" (UID: \"9a930808-b419-4475-8c2e-c13d888236f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh" Nov 03 20:30:54.775574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.775462 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a930808-b419-4475-8c2e-c13d888236f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6hwjh\" (UID: \"9a930808-b419-4475-8c2e-c13d888236f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh" Nov 03 20:30:54.776220 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.776201 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a930808-b419-4475-8c2e-c13d888236f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6hwjh\" (UID: \"9a930808-b419-4475-8c2e-c13d888236f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh" Nov 03 20:30:54.777826 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.777800 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a930808-b419-4475-8c2e-c13d888236f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6hwjh\" (UID: \"9a930808-b419-4475-8c2e-c13d888236f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh" Nov 03 20:30:54.783733 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.783710 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqsz7\" (UniqueName: \"kubernetes.io/projected/9a930808-b419-4475-8c2e-c13d888236f6-kube-api-access-jqsz7\") pod \"maas-api-revoke-sub-a4aef7d7-6hwjh\" (UID: \"9a930808-b419-4475-8c2e-c13d888236f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh" Nov 03 20:30:54.897503 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:54.897466 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh" Nov 03 20:30:55.176270 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:55.176238 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh"] Nov 03 20:30:55.177347 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:30:55.177320 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a930808_b419_4475_8c2e_c13d888236f6.slice/crio-61f7a788abeca0a87c31b9322520a78b7c4d6eb85770d045a2ef8030294529c1 WatchSource:0}: Error finding container 61f7a788abeca0a87c31b9322520a78b7c4d6eb85770d045a2ef8030294529c1: Status 404 returned error can't find the container with id 61f7a788abeca0a87c31b9322520a78b7c4d6eb85770d045a2ef8030294529c1 Nov 03 20:30:55.987004 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:55.986965 5111 generic.go:356] "Generic (PLEG): container finished" podID="9a930808-b419-4475-8c2e-c13d888236f6" containerID="8ce0fe84bccf04990c74e5167f3c8bcc8523896d4f305c47d641d622af410d5c" exitCode=0 Nov 03 20:30:55.987384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:55.987048 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh" event={"ID":"9a930808-b419-4475-8c2e-c13d888236f6","Type":"ContainerDied","Data":"8ce0fe84bccf04990c74e5167f3c8bcc8523896d4f305c47d641d622af410d5c"} Nov 03 20:30:55.987384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:55.987107 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh" event={"ID":"9a930808-b419-4475-8c2e-c13d888236f6","Type":"ContainerStarted","Data":"61f7a788abeca0a87c31b9322520a78b7c4d6eb85770d045a2ef8030294529c1"} Nov 03 20:30:57.137461 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:57.137434 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh" Nov 03 20:30:57.198792 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:57.198751 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a930808-b419-4475-8c2e-c13d888236f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a930808-b419-4475-8c2e-c13d888236f6-maas-api-xks-ca\") pod \"9a930808-b419-4475-8c2e-c13d888236f6\" (UID: \"9a930808-b419-4475-8c2e-c13d888236f6\") " Nov 03 20:30:57.199000 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:57.198928 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a930808-b419-4475-8c2e-c13d888236f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a930808-b419-4475-8c2e-c13d888236f6-maas-api-openshift-ca\") pod \"9a930808-b419-4475-8c2e-c13d888236f6\" (UID: \"9a930808-b419-4475-8c2e-c13d888236f6\") " Nov 03 20:30:57.199000 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:57.198971 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a930808-b419-4475-8c2e-c13d888236f6-kube-api-access-jqsz7\" (UniqueName: \"kubernetes.io/projected/9a930808-b419-4475-8c2e-c13d888236f6-kube-api-access-jqsz7\") pod \"9a930808-b419-4475-8c2e-c13d888236f6\" (UID: \"9a930808-b419-4475-8c2e-c13d888236f6\") " Nov 03 20:30:57.199292 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:57.199265 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a930808-b419-4475-8c2e-c13d888236f6-maas-api-openshift-ca" pod "9a930808-b419-4475-8c2e-c13d888236f6" (UID: "9a930808-b419-4475-8c2e-c13d888236f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:57.200942 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:57.200920 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a930808-b419-4475-8c2e-c13d888236f6-maas-api-xks-ca" pod "9a930808-b419-4475-8c2e-c13d888236f6" (UID: "9a930808-b419-4475-8c2e-c13d888236f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:57.201054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:57.201042 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a930808-b419-4475-8c2e-c13d888236f6-kube-api-access-jqsz7" pod "9a930808-b419-4475-8c2e-c13d888236f6" (UID: "9a930808-b419-4475-8c2e-c13d888236f6"). InnerVolumeSpecName "kube-api-access-jqsz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:57.300558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:57.300446 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a930808-b419-4475-8c2e-c13d888236f6-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:57.300558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:57.300479 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a930808-b419-4475-8c2e-c13d888236f6-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:57.300558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:57.300490 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqsz7\" (UniqueName: \"kubernetes.io/projected/9a930808-b419-4475-8c2e-c13d888236f6-kube-api-access-jqsz7\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:58.001457 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:58.001418 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh" event={"ID":"9a930808-b419-4475-8c2e-c13d888236f6","Type":"ContainerDied","Data":"61f7a788abeca0a87c31b9322520a78b7c4d6eb85770d045a2ef8030294529c1"} Nov 03 20:30:58.001679 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:58.001483 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61f7a788abeca0a87c31b9322520a78b7c4d6eb85770d045a2ef8030294529c1" Nov 03 20:30:58.001679 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:30:58.001428 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh" Nov 03 20:31:05.057321 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:31:05.057282 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:31:12.026738 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:12.026696 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz"] Nov 03 20:31:12.029374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:12.029341 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e63e2422-2zcbz"] Nov 03 20:31:12.180712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:12.180678 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="873efceb-087b-4442-966e-e395b73ae3c5" path="/var/lib/kubelet/pods/873efceb-087b-4442-966e-e395b73ae3c5/volumes" Nov 03 20:31:13.236519 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.236430 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd"] Nov 03 20:31:13.236979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.236961 5111 cpu_manager.go:414] "containerMap: removing container" podUID="9a930808-b419-4475-8c2e-c13d888236f6" containerName="revoke-keys" Nov 03 20:31:13.237027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.236980 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a930808-b419-4475-8c2e-c13d888236f6" containerName="revoke-keys" Nov 03 20:31:13.237088 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.237079 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a930808-b419-4475-8c2e-c13d888236f6" containerName="revoke-keys" Nov 03 20:31:13.240635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.240616 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd" Nov 03 20:31:13.245342 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.245311 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:31:13.245496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.245321 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:31:13.245496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.245368 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:31:13.245496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.245389 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:31:13.249609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.249580 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd"] Nov 03 20:31:13.365766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.365721 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hsljd\" (UID: \"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd" Nov 03 20:31:13.365766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.365766 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpvks\" (UniqueName: \"kubernetes.io/projected/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-kube-api-access-fpvks\") pod \"maas-api-revoke-sub-79805ca9-hsljd\" (UID: \"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd" Nov 03 20:31:13.366008 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.365788 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hsljd\" (UID: \"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd" Nov 03 20:31:13.466808 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.466773 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hsljd\" (UID: \"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd" Nov 03 20:31:13.466808 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.466812 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpvks\" (UniqueName: \"kubernetes.io/projected/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-kube-api-access-fpvks\") pod \"maas-api-revoke-sub-79805ca9-hsljd\" (UID: \"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd" Nov 03 20:31:13.467050 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.466837 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hsljd\" (UID: \"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd" Nov 03 20:31:13.467563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.467522 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hsljd\" (UID: \"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd" Nov 03 20:31:13.469210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.469185 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hsljd\" (UID: \"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd" Nov 03 20:31:13.475120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.475092 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpvks\" (UniqueName: \"kubernetes.io/projected/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-kube-api-access-fpvks\") pod \"maas-api-revoke-sub-79805ca9-hsljd\" (UID: \"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd" Nov 03 20:31:13.561313 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.561225 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd" Nov 03 20:31:13.837988 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:13.837955 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd"] Nov 03 20:31:13.839544 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:31:13.839506 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bca32e3_4a0f_4955_86ac_ff6cf8321ce9.slice/crio-74d27cda3d1011add4cbd52ea668fcb7531f3702b6045a164b028411ed92bafb WatchSource:0}: Error finding container 74d27cda3d1011add4cbd52ea668fcb7531f3702b6045a164b028411ed92bafb: Status 404 returned error can't find the container with id 74d27cda3d1011add4cbd52ea668fcb7531f3702b6045a164b028411ed92bafb Nov 03 20:31:14.149642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:14.149604 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd" event={"ID":"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9","Type":"ContainerStarted","Data":"74d27cda3d1011add4cbd52ea668fcb7531f3702b6045a164b028411ed92bafb"} Nov 03 20:31:15.158011 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:15.157976 5111 generic.go:356] "Generic (PLEG): container finished" podID="8bca32e3-4a0f-4955-86ac-ff6cf8321ce9" containerID="7708cee7cdd89388b0405ba9729e9a711b739529673143592172d0183127d884" exitCode=0 Nov 03 20:31:15.158434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:15.158067 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd" event={"ID":"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9","Type":"ContainerDied","Data":"7708cee7cdd89388b0405ba9729e9a711b739529673143592172d0183127d884"} Nov 03 20:31:16.313474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:16.313447 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd" Nov 03 20:31:16.394873 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:16.394819 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-maas-api-xks-ca\") pod \"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9\" (UID: \"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9\") " Nov 03 20:31:16.395090 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:16.394947 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-maas-api-openshift-ca\") pod \"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9\" (UID: \"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9\") " Nov 03 20:31:16.395090 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:16.395025 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-kube-api-access-fpvks\" (UniqueName: \"kubernetes.io/projected/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-kube-api-access-fpvks\") pod \"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9\" (UID: \"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9\") " Nov 03 20:31:16.395309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:16.395284 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-maas-api-openshift-ca" pod "8bca32e3-4a0f-4955-86ac-ff6cf8321ce9" (UID: "8bca32e3-4a0f-4955-86ac-ff6cf8321ce9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:16.395407 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:16.395395 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:16.397116 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:16.397090 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-kube-api-access-fpvks" pod "8bca32e3-4a0f-4955-86ac-ff6cf8321ce9" (UID: "8bca32e3-4a0f-4955-86ac-ff6cf8321ce9"). InnerVolumeSpecName "kube-api-access-fpvks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:16.397183 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:16.397138 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-maas-api-xks-ca" pod "8bca32e3-4a0f-4955-86ac-ff6cf8321ce9" (UID: "8bca32e3-4a0f-4955-86ac-ff6cf8321ce9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:16.497021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:16.496912 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpvks\" (UniqueName: \"kubernetes.io/projected/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-kube-api-access-fpvks\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:16.497021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:16.496965 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:17.175992 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:17.175961 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd" Nov 03 20:31:17.175992 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:17.175972 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd" event={"ID":"8bca32e3-4a0f-4955-86ac-ff6cf8321ce9","Type":"ContainerDied","Data":"74d27cda3d1011add4cbd52ea668fcb7531f3702b6045a164b028411ed92bafb"} Nov 03 20:31:17.176278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:17.176009 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74d27cda3d1011add4cbd52ea668fcb7531f3702b6045a164b028411ed92bafb" Nov 03 20:31:21.022600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:21.022523 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr"] Nov 03 20:31:21.025314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:21.025275 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7907172-fhcqr"] Nov 03 20:31:22.181082 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:22.181047 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b041b8d-92e6-488e-809c-40cfa2758f3e" path="/var/lib/kubelet/pods/0b041b8d-92e6-488e-809c-40cfa2758f3e/volumes" Nov 03 20:31:28.027645 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:28.027589 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj"] Nov 03 20:31:28.029635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:28.029606 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2"] Nov 03 20:31:28.032020 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:28.031995 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3a20436-kkptj"] Nov 03 20:31:28.034094 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:28.034060 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5668248f-862c2"] Nov 03 20:31:28.214014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:28.213983 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c26fcf44-2799-4302-9dc8-84f8a3b389eb" path="/var/lib/kubelet/pods/c26fcf44-2799-4302-9dc8-84f8a3b389eb/volumes" Nov 03 20:31:28.214760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:28.214735 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2d63738-ed89-4361-a251-f4fa9bd46253" path="/var/lib/kubelet/pods/d2d63738-ed89-4361-a251-f4fa9bd46253/volumes" Nov 03 20:31:35.056658 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:31:35.056613 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:31:38.465560 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:38.465497 5111 scope.go:122] "RemoveContainer" containerID="27885f6751f3869220e3a5ad8a31a8e7d0ff2cf8338357576c46d1a6db4587e7" Nov 03 20:31:38.475396 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:38.475369 5111 scope.go:122] "RemoveContainer" containerID="e41bcf9aa4d1370f42e4c4c9aa49707111174c97d2edef070c5ed1bee30d4d2e" Nov 03 20:31:38.484462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:38.484440 5111 scope.go:122] "RemoveContainer" containerID="d161f4b844a1bf1424c00e9e941685142a22f0094ebbbc0ddbe9e24fd0b9588d" Nov 03 20:31:38.493157 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:38.493136 5111 scope.go:122] "RemoveContainer" containerID="185c24c4cd92518ae95dd98b63640701767bae3f5744031b59db94308a55fc1b" Nov 03 20:31:38.501601 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:38.501581 5111 scope.go:122] "RemoveContainer" containerID="ed912031179f70e1612c06904f09c00f1466848173471470aa2b3c12d1651d5d" Nov 03 20:31:38.511708 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:38.511682 5111 scope.go:122] "RemoveContainer" containerID="d23806f37e27ba53a3951c2ca07b09aa2d614ca4452fa1e651ad5419c4ec58d5" Nov 03 20:31:38.520232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:38.520213 5111 scope.go:122] "RemoveContainer" containerID="fdfafd3471c520281796e3f08b43fa222014e8d43ab79bd06fd1e920c59e7049" Nov 03 20:31:38.528878 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:38.528855 5111 scope.go:122] "RemoveContainer" containerID="48d4a683a2677878859d64ba5446aad957d6f45333c86caab40d425fdbd1f530" Nov 03 20:31:38.539076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:38.539058 5111 scope.go:122] "RemoveContainer" containerID="d56c36f7efa98a54048647c8a04adf84e32ed877f79901dfe07ab3656883a1e1" Nov 03 20:31:38.547709 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:38.547690 5111 scope.go:122] "RemoveContainer" containerID="31b53ee687201111c2542f7e1bf235950479c4867c4cfa5119935c36117e115e" Nov 03 20:31:51.023906 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:51.023865 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn"] Nov 03 20:31:51.028326 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:51.028289 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7"] Nov 03 20:31:51.030452 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:51.030425 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e03b1fa6-68vqn"] Nov 03 20:31:51.032399 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:51.032374 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f57365-j4tb7"] Nov 03 20:31:52.181401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:52.181366 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd1beb8-21b5-49f9-a228-e2ddf280fc48" path="/var/lib/kubelet/pods/5dd1beb8-21b5-49f9-a228-e2ddf280fc48/volumes" Nov 03 20:31:52.181865 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:52.181849 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad12f36-4946-4027-a489-7a7f4b6b205b" path="/var/lib/kubelet/pods/fad12f36-4946-4027-a489-7a7f4b6b205b/volumes" Nov 03 20:31:58.024767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:58.024728 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q"] Nov 03 20:31:58.027385 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:58.027348 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ca16b1-6wg6q"] Nov 03 20:31:58.029575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:58.029548 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz"] Nov 03 20:31:58.031628 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:58.031604 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14290fa9-4z2bz"] Nov 03 20:31:58.181949 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:58.181914 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77848c07-7655-4d10-9a19-76e07bc77e8a" path="/var/lib/kubelet/pods/77848c07-7655-4d10-9a19-76e07bc77e8a/volumes" Nov 03 20:31:58.182395 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:31:58.182381 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc60303a-7c35-40e2-a552-4ddf22fa666f" path="/var/lib/kubelet/pods/bc60303a-7c35-40e2-a552-4ddf22fa666f/volumes" Nov 03 20:32:01.027116 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:01.027077 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd"] Nov 03 20:32:01.030608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:01.030577 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a64755c9-sgtfd"] Nov 03 20:32:02.181173 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:02.181145 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e0c2bf-68f5-4cbd-a110-c24180dd4525" path="/var/lib/kubelet/pods/76e0c2bf-68f5-4cbd-a110-c24180dd4525/volumes" Nov 03 20:32:05.057294 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:32:05.057249 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:32:23.421557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:23.420625 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gmqdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:32:23.421557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:23.420706 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" podUID="cf6540f3-53cd-4088-bf2e-e44670ad75b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:23.450669 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:23.450632 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h7vtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:32:23.450949 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:23.450916 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" podUID="d969be31-5822-40ef-be6f-c6627bfa5956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:23.459639 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:23.459610 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4vnbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:32:23.459855 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:23.459829 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" podUID="cf5856f8-f452-4f05-b083-fb88f3db028a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:28.011006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.010968 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm"] Nov 03 20:32:28.013348 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.011218 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" podUID="7d0a532e-8921-486e-ad83-2cb3c1e72292" containerName="discovery" containerID="cri-o://b0c371317c1fea9528ae424c265322c2c66e962a6528c1a93b03014210a9da0f" gracePeriod=30 Nov 03 20:32:28.297252 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.297220 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:32:28.448042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.448003 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-token\" (UniqueName: \"kubernetes.io/projected/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-token\") pod \"7d0a532e-8921-486e-ad83-2cb3c1e72292\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " Nov 03 20:32:28.448258 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.448095 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-cacerts\" (UniqueName: \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-cacerts\") pod \"7d0a532e-8921-486e-ad83-2cb3c1e72292\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " Nov 03 20:32:28.448258 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.448220 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-kubeconfig\") pod \"7d0a532e-8921-486e-ad83-2cb3c1e72292\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " Nov 03 20:32:28.448385 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.448263 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0a532e-8921-486e-ad83-2cb3c1e72292-kube-api-access-5jfkp\" (UniqueName: \"kubernetes.io/projected/7d0a532e-8921-486e-ad83-2cb3c1e72292-kube-api-access-5jfkp\") pod \"7d0a532e-8921-486e-ad83-2cb3c1e72292\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " Nov 03 20:32:28.448385 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.448299 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-csr-dns-cert\") pod \"7d0a532e-8921-486e-ad83-2cb3c1e72292\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " Nov 03 20:32:28.448385 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.448357 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d0a532e-8921-486e-ad83-2cb3c1e72292-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d0a532e-8921-486e-ad83-2cb3c1e72292-local-certs\") pod \"7d0a532e-8921-486e-ad83-2cb3c1e72292\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " Nov 03 20:32:28.448579 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.448407 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-csr-ca-configmap\") pod \"7d0a532e-8921-486e-ad83-2cb3c1e72292\" (UID: \"7d0a532e-8921-486e-ad83-2cb3c1e72292\") " Nov 03 20:32:28.449203 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.449153 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-csr-ca-configmap" pod "7d0a532e-8921-486e-ad83-2cb3c1e72292" (UID: "7d0a532e-8921-486e-ad83-2cb3c1e72292"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:28.450908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.450871 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-token" pod "7d0a532e-8921-486e-ad83-2cb3c1e72292" (UID: "7d0a532e-8921-486e-ad83-2cb3c1e72292"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:28.451140 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.451101 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0a532e-8921-486e-ad83-2cb3c1e72292-local-certs" pod "7d0a532e-8921-486e-ad83-2cb3c1e72292" (UID: "7d0a532e-8921-486e-ad83-2cb3c1e72292"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:28.451140 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.451116 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-csr-dns-cert" pod "7d0a532e-8921-486e-ad83-2cb3c1e72292" (UID: "7d0a532e-8921-486e-ad83-2cb3c1e72292"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:28.451302 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.451133 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-cacerts" pod "7d0a532e-8921-486e-ad83-2cb3c1e72292" (UID: "7d0a532e-8921-486e-ad83-2cb3c1e72292"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:28.451390 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.451369 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0a532e-8921-486e-ad83-2cb3c1e72292-kube-api-access-5jfkp" pod "7d0a532e-8921-486e-ad83-2cb3c1e72292" (UID: "7d0a532e-8921-486e-ad83-2cb3c1e72292"). InnerVolumeSpecName "kube-api-access-5jfkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:28.452024 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.451935 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-kubeconfig" pod "7d0a532e-8921-486e-ad83-2cb3c1e72292" (UID: "7d0a532e-8921-486e-ad83-2cb3c1e72292"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:28.549780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.549694 5111 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d0a532e-8921-486e-ad83-2cb3c1e72292-local-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:28.549780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.549726 5111 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-csr-ca-configmap\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:28.549780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.549741 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:28.549780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.549750 5111 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-cacerts\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:28.549780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.549761 5111 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-kubeconfig\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:28.549780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.549769 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jfkp\" (UniqueName: \"kubernetes.io/projected/7d0a532e-8921-486e-ad83-2cb3c1e72292-kube-api-access-5jfkp\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:28.549780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.549778 5111 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7d0a532e-8921-486e-ad83-2cb3c1e72292-istio-csr-dns-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:28.757309 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.757271 5111 generic.go:356] "Generic (PLEG): container finished" podID="7d0a532e-8921-486e-ad83-2cb3c1e72292" containerID="b0c371317c1fea9528ae424c265322c2c66e962a6528c1a93b03014210a9da0f" exitCode=0 Nov 03 20:32:28.757496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.757317 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" Nov 03 20:32:28.757496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.757359 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" event={"ID":"7d0a532e-8921-486e-ad83-2cb3c1e72292","Type":"ContainerDied","Data":"b0c371317c1fea9528ae424c265322c2c66e962a6528c1a93b03014210a9da0f"} Nov 03 20:32:28.757496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.757414 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm" event={"ID":"7d0a532e-8921-486e-ad83-2cb3c1e72292","Type":"ContainerDied","Data":"444f097ce7af8bc8bc3333089ecc424ebe4745134bc0bce4e800d147a50f75f7"} Nov 03 20:32:28.757496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.757437 5111 scope.go:122] "RemoveContainer" containerID="b0c371317c1fea9528ae424c265322c2c66e962a6528c1a93b03014210a9da0f" Nov 03 20:32:28.769457 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.769438 5111 scope.go:122] "RemoveContainer" containerID="b0c371317c1fea9528ae424c265322c2c66e962a6528c1a93b03014210a9da0f" Nov 03 20:32:28.769755 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:32:28.769736 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c371317c1fea9528ae424c265322c2c66e962a6528c1a93b03014210a9da0f\": container with ID starting with b0c371317c1fea9528ae424c265322c2c66e962a6528c1a93b03014210a9da0f not found: ID does not exist" containerID="b0c371317c1fea9528ae424c265322c2c66e962a6528c1a93b03014210a9da0f" Nov 03 20:32:28.769815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.769764 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c371317c1fea9528ae424c265322c2c66e962a6528c1a93b03014210a9da0f"} err="failed to get container status \"b0c371317c1fea9528ae424c265322c2c66e962a6528c1a93b03014210a9da0f\": rpc error: code = NotFound desc = could not find container \"b0c371317c1fea9528ae424c265322c2c66e962a6528c1a93b03014210a9da0f\": container with ID starting with b0c371317c1fea9528ae424c265322c2c66e962a6528c1a93b03014210a9da0f not found: ID does not exist" Nov 03 20:32:28.779900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.779869 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm"] Nov 03 20:32:28.787612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:28.787587 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vk8mm"] Nov 03 20:32:30.027980 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:30.027945 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq"] Nov 03 20:32:30.035022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:30.034995 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47da3855-5gbwq"] Nov 03 20:32:30.181518 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:30.181483 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d7ffd96-9659-4080-b6bc-9343be470073" path="/var/lib/kubelet/pods/1d7ffd96-9659-4080-b6bc-9343be470073/volumes" Nov 03 20:32:30.181967 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:30.181952 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d0a532e-8921-486e-ad83-2cb3c1e72292" path="/var/lib/kubelet/pods/7d0a532e-8921-486e-ad83-2cb3c1e72292/volumes" Nov 03 20:32:35.056754 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:32:35.056711 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:32:38.451863 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.451826 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb"] Nov 03 20:32:38.452267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.452249 5111 cpu_manager.go:414] "containerMap: removing container" podUID="7d0a532e-8921-486e-ad83-2cb3c1e72292" containerName="discovery" Nov 03 20:32:38.452267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.452260 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0a532e-8921-486e-ad83-2cb3c1e72292" containerName="discovery" Nov 03 20:32:38.452351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.452279 5111 cpu_manager.go:414] "containerMap: removing container" podUID="8bca32e3-4a0f-4955-86ac-ff6cf8321ce9" containerName="revoke-keys" Nov 03 20:32:38.452351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.452284 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bca32e3-4a0f-4955-86ac-ff6cf8321ce9" containerName="revoke-keys" Nov 03 20:32:38.452422 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.452376 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d0a532e-8921-486e-ad83-2cb3c1e72292" containerName="discovery" Nov 03 20:32:38.452422 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.452383 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bca32e3-4a0f-4955-86ac-ff6cf8321ce9" containerName="revoke-keys" Nov 03 20:32:38.455563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.455544 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" Nov 03 20:32:38.461972 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.461934 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:32:38.462767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.462734 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:32:38.462943 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.462918 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:32:38.463082 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.463012 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:32:38.463779 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.463711 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb"] Nov 03 20:32:38.538856 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.538823 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-t4xvb\" (UID: \"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" Nov 03 20:32:38.539030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.538871 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-t4xvb\" (UID: \"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" Nov 03 20:32:38.539030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.538904 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkvqh\" (UniqueName: \"kubernetes.io/projected/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-kube-api-access-pkvqh\") pod \"maas-api-revoke-sub-5a29efb3-t4xvb\" (UID: \"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" Nov 03 20:32:38.616809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.616781 5111 scope.go:122] "RemoveContainer" containerID="d373c09ca3881198a3a587aadc6b9da7994bc4a22ff41141fc56b770f127c62a" Nov 03 20:32:38.626179 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.626155 5111 scope.go:122] "RemoveContainer" containerID="2e62c042c5e70ab2816162bf7e62d1affb98e13c9066cde3507437400c4d1e93" Nov 03 20:32:38.634980 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.634960 5111 scope.go:122] "RemoveContainer" containerID="5dc0dba587c8b92f4e91c6640c70ee9623857f1b82bfd6001753c0b73203233e" Nov 03 20:32:38.640390 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.640360 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-t4xvb\" (UID: \"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" Nov 03 20:32:38.640482 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.640421 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-t4xvb\" (UID: \"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" Nov 03 20:32:38.640482 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.640465 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkvqh\" (UniqueName: \"kubernetes.io/projected/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-kube-api-access-pkvqh\") pod \"maas-api-revoke-sub-5a29efb3-t4xvb\" (UID: \"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" Nov 03 20:32:38.641031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.641008 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-t4xvb\" (UID: \"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" Nov 03 20:32:38.642763 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.642741 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-t4xvb\" (UID: \"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" Nov 03 20:32:38.644111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.644094 5111 scope.go:122] "RemoveContainer" containerID="641e17b3fbb901df9bef5c82e6d06aa6e1cd57a287876a843717da88f93e6a3f" Nov 03 20:32:38.648168 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.648149 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkvqh\" (UniqueName: \"kubernetes.io/projected/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-kube-api-access-pkvqh\") pod \"maas-api-revoke-sub-5a29efb3-t4xvb\" (UID: \"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" Nov 03 20:32:38.667056 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.667032 5111 scope.go:122] "RemoveContainer" containerID="b79952d3db616b5c53d16e6c9920607d29c9afa644fdd2e3a657eb74ca2bacde" Nov 03 20:32:38.676391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.676365 5111 scope.go:122] "RemoveContainer" containerID="731328f80df547b67c70f18d464eaa344a28d2bb0a6d5148155308ed44eb4c0a" Nov 03 20:32:38.684941 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.684922 5111 scope.go:122] "RemoveContainer" containerID="6e984e87444e88b4bb9ad48fac3dc470f9bf319cf62b8b0e04fecca240f8a7f4" Nov 03 20:32:38.693499 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.693478 5111 scope.go:122] "RemoveContainer" containerID="fc56f64b104ee650e80a41f60ca292e50dc0bd19369aeb275208c471f205715f" Nov 03 20:32:38.701965 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.701911 5111 scope.go:122] "RemoveContainer" containerID="26543d232f342f8f6d18c7cccf4601473bdaa8ca8528076aafdb48f87ed4e1a1" Nov 03 20:32:38.710263 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.710241 5111 scope.go:122] "RemoveContainer" containerID="4d6eef0d4fe65c38a2ad67f4524e7db56912fb6c4fc52eed9a875ac67776862b" Nov 03 20:32:38.718873 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.718855 5111 scope.go:122] "RemoveContainer" containerID="c02b619fc09eb6cedd9b240e63423df6af05c2c62823bde50046e8bd3b439f68" Nov 03 20:32:38.727314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.727296 5111 scope.go:122] "RemoveContainer" containerID="2a4ef095be3a6770ff116ea63c76e696ba321c8e5fef21814e76103d62d2f7aa" Nov 03 20:32:38.735556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.735522 5111 scope.go:122] "RemoveContainer" containerID="daa367ca3bf93e4fdbf3733d0c9b14c2d60e3dc3e109d1c731d1c87f077d8275" Nov 03 20:32:38.743950 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.743929 5111 scope.go:122] "RemoveContainer" containerID="f2f0c8579dab16dcd9f0332d0947401fa84bd99fbece61b45ad9e72b39fc5728" Nov 03 20:32:38.751948 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.751931 5111 scope.go:122] "RemoveContainer" containerID="ced7f9352f3ef9133742a6feb6aad35b7efda32886c928ea34415177cae1b8f2" Nov 03 20:32:38.771274 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:38.771250 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" Nov 03 20:32:39.061908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:39.061880 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb"] Nov 03 20:32:39.064230 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:32:39.064203 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d3f5fd_69f3_4a98_8343_49a6dd2f3fef.slice/crio-f8367efba4be9f6efde382a9ea323ddb45145e5e566fd0ee2ebb81032810a61c WatchSource:0}: Error finding container f8367efba4be9f6efde382a9ea323ddb45145e5e566fd0ee2ebb81032810a61c: Status 404 returned error can't find the container with id f8367efba4be9f6efde382a9ea323ddb45145e5e566fd0ee2ebb81032810a61c Nov 03 20:32:39.841594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:39.841523 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" event={"ID":"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef","Type":"ContainerStarted","Data":"a72f8982bb196703e5ad67679909b78666411908d1eaf96f7d6a3a6d14ecfe7f"} Nov 03 20:32:39.842162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:39.841616 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" event={"ID":"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef","Type":"ContainerStarted","Data":"f8367efba4be9f6efde382a9ea323ddb45145e5e566fd0ee2ebb81032810a61c"} Nov 03 20:32:39.859051 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:39.858987 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" podStartSLOduration=1.8589698000000001 podStartE2EDuration="1.8589698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:39.858485388 +0000 UTC m=+3189.820863827" watchObservedRunningTime="2026-11-03 20:32:39.8589698 +0000 UTC m=+3189.821348235" Nov 03 20:32:40.897189 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:40.896638 5111 generic.go:356] "Generic (PLEG): container finished" podID="52d3f5fd-69f3-4a98-8343-49a6dd2f3fef" containerID="a72f8982bb196703e5ad67679909b78666411908d1eaf96f7d6a3a6d14ecfe7f" exitCode=0 Nov 03 20:32:40.897189 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:40.896821 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" event={"ID":"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef","Type":"ContainerDied","Data":"a72f8982bb196703e5ad67679909b78666411908d1eaf96f7d6a3a6d14ecfe7f"} Nov 03 20:32:42.058961 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:42.058935 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" Nov 03 20:32:42.118678 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:42.118645 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-maas-api-xks-ca\") pod \"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef\" (UID: \"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef\") " Nov 03 20:32:42.118854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:42.118695 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-maas-api-openshift-ca\") pod \"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef\" (UID: \"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef\") " Nov 03 20:32:42.118854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:42.118775 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-kube-api-access-pkvqh\" (UniqueName: \"kubernetes.io/projected/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-kube-api-access-pkvqh\") pod \"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef\" (UID: \"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef\") " Nov 03 20:32:42.119110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:42.119083 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-maas-api-openshift-ca" pod "52d3f5fd-69f3-4a98-8343-49a6dd2f3fef" (UID: "52d3f5fd-69f3-4a98-8343-49a6dd2f3fef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:42.120871 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:42.120851 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-maas-api-xks-ca" pod "52d3f5fd-69f3-4a98-8343-49a6dd2f3fef" (UID: "52d3f5fd-69f3-4a98-8343-49a6dd2f3fef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:42.120978 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:42.120960 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-kube-api-access-pkvqh" pod "52d3f5fd-69f3-4a98-8343-49a6dd2f3fef" (UID: "52d3f5fd-69f3-4a98-8343-49a6dd2f3fef"). InnerVolumeSpecName "kube-api-access-pkvqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:42.220430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:42.220396 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:42.220430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:42.220430 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkvqh\" (UniqueName: \"kubernetes.io/projected/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-kube-api-access-pkvqh\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:42.220669 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:42.220444 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:42.912492 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:42.912455 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" event={"ID":"52d3f5fd-69f3-4a98-8343-49a6dd2f3fef","Type":"ContainerDied","Data":"f8367efba4be9f6efde382a9ea323ddb45145e5e566fd0ee2ebb81032810a61c"} Nov 03 20:32:42.912703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:42.912498 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb" Nov 03 20:32:42.912703 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:32:42.912511 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8367efba4be9f6efde382a9ea323ddb45145e5e566fd0ee2ebb81032810a61c" Nov 03 20:33:05.057149 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:33:05.057082 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:33:17.027049 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:33:17.027003 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8"] Nov 03 20:33:17.029940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:33:17.029908 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8jht8"] Nov 03 20:33:18.181778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:33:18.181744 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd" path="/var/lib/kubelet/pods/a9c3dda0-fde6-4c15-bf6f-323f4c0a7ccd/volumes" Nov 03 20:33:35.062083 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:33:35.062034 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:33:38.853803 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:33:38.853770 5111 scope.go:122] "RemoveContainer" containerID="6df9e12e65d8ec659ccd9d21d74cff4d45501e8b74560469fcc47a7abfdb37bc" Nov 03 20:33:38.863246 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:33:38.863219 5111 scope.go:122] "RemoveContainer" containerID="c24656c020f30c09ad0a7d1fbb18af7e9274dde774b1ac3fe155ad246d05d29d" Nov 03 20:33:38.871650 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:33:38.871627 5111 scope.go:122] "RemoveContainer" containerID="2462ba8a187fba6a55ea84ae8feed52eb3cf9da5ed7f34551b9ba0e800b338cc" Nov 03 20:33:38.879635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:33:38.879616 5111 scope.go:122] "RemoveContainer" containerID="3586e96c785bf27c5192f9f79ebe5408d255153080836c9bd75a6af59650ca06" Nov 03 20:33:38.887508 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:33:38.887487 5111 scope.go:122] "RemoveContainer" containerID="194b2c902a87e92d98d865c7423c5e31cb6df9c7fb764e513fc867a2cb9fe51f" Nov 03 20:33:38.895242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:33:38.895226 5111 scope.go:122] "RemoveContainer" containerID="fac5c40e8439fed67cce1c68d3c2d92691558723c66cf6b59f9494c8bf934012" Nov 03 20:33:38.902951 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:33:38.902932 5111 scope.go:122] "RemoveContainer" containerID="00d05397f20e9bd989209a3ff9b407868b5bb14d0698f78bdb2c7a604c5c9274" Nov 03 20:33:38.910981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:33:38.910963 5111 scope.go:122] "RemoveContainer" containerID="c483b88f4ae76b281dc5583ea64d1df1897fb8b2c0f00527101874fdc3711bc9" Nov 03 20:33:38.919725 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:33:38.919702 5111 scope.go:122] "RemoveContainer" containerID="f53d8d533fa6ead5451ac4a7e6a853f223a1893ba2c0fb6652dc7e138f009ce9" Nov 03 20:34:05.057354 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:34:05.057311 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:34:13.020667 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:34:13.020584 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g"] Nov 03 20:34:13.024936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:34:13.024906 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vls8g"] Nov 03 20:34:14.181546 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:34:14.181506 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a27dec6-8bae-4276-8255-4acdb766880b" path="/var/lib/kubelet/pods/9a27dec6-8bae-4276-8255-4acdb766880b/volumes" Nov 03 20:34:30.970637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:34:30.970608 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:34:30.976208 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:34:30.976181 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:34:30.976395 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:34:30.976375 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:34:30.981975 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:34:30.981955 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:34:32.020582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:34:32.020546 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4"] Nov 03 20:34:32.023772 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:34:32.023747 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ghcn4"] Nov 03 20:34:32.181729 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:34:32.181694 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d61b606-c574-4963-b13e-669efa986122" path="/var/lib/kubelet/pods/6d61b606-c574-4963-b13e-669efa986122/volumes" Nov 03 20:34:35.056716 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:34:35.056683 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:34:38.973505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:34:38.973471 5111 scope.go:122] "RemoveContainer" containerID="6bd878f23050ffe12b78ca473c2c6c4fef6c5c55ec8ba7baa2f5f06d401eaff1" Nov 03 20:34:43.022769 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:34:43.022721 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d"] Nov 03 20:34:43.024927 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:34:43.024890 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k9h5d"] Nov 03 20:34:44.181621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:34:44.181590 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc43082d-8921-4f7a-b1a1-5e2336501117" path="/var/lib/kubelet/pods/fc43082d-8921-4f7a-b1a1-5e2336501117/volumes" Nov 03 20:35:05.056333 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:35:05.056292 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:35:35.056051 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:35:35.056009 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:35:39.000060 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:35:39.000027 5111 scope.go:122] "RemoveContainer" containerID="80b8a5d840778d12ce7b19f11f8366d62beec8ee018f54f575dab8471868d7f3" Nov 03 20:35:39.008914 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:35:39.008893 5111 scope.go:122] "RemoveContainer" containerID="08c2bc39a68ba143752657c97262589f3f59d114a857d9779d05464af1781f4f" Nov 03 20:35:40.021738 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:35:40.021692 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm"] Nov 03 20:35:40.024419 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:35:40.024391 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-p5dwm"] Nov 03 20:35:40.182027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:35:40.181989 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bba6ee8-1f32-43bc-9da6-345fb14a9011" path="/var/lib/kubelet/pods/1bba6ee8-1f32-43bc-9da6-345fb14a9011/volumes" Nov 03 20:35:58.019645 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:35:58.019607 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh"] Nov 03 20:35:58.023149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:35:58.023121 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6hwjh"] Nov 03 20:35:58.181670 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:35:58.181635 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a930808-b419-4475-8c2e-c13d888236f6" path="/var/lib/kubelet/pods/9a930808-b419-4475-8c2e-c13d888236f6/volumes" Nov 03 20:36:05.055652 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:36:05.055610 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:36:17.031026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:36:17.030985 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd"] Nov 03 20:36:17.040989 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:36:17.040954 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hsljd"] Nov 03 20:36:18.182321 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:36:18.182281 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bca32e3-4a0f-4955-86ac-ff6cf8321ce9" path="/var/lib/kubelet/pods/8bca32e3-4a0f-4955-86ac-ff6cf8321ce9/volumes" Nov 03 20:36:35.056882 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:36:35.056843 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:36:39.034341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:36:39.034308 5111 scope.go:122] "RemoveContainer" containerID="9875c99d590bfc5ed230d80d065d2cddfed6709f1bd15940d0607ab5ff3d203d" Nov 03 20:36:39.043939 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:36:39.043913 5111 scope.go:122] "RemoveContainer" containerID="f911a6fb1404e50715c182d6345b4117e77bc2c6bf1ce1a60b70c5175668dd60" Nov 03 20:37:05.055595 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:37:05.055555 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:37:35.056444 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:37:35.056401 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:37:39.072834 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:39.072801 5111 scope.go:122] "RemoveContainer" containerID="7708cee7cdd89388b0405ba9729e9a711b739529673143592172d0183127d884" Nov 03 20:37:39.081837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:39.081816 5111 scope.go:122] "RemoveContainer" containerID="8ce0fe84bccf04990c74e5167f3c8bcc8523896d4f305c47d641d622af410d5c" Nov 03 20:37:42.020347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:42.020308 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb"] Nov 03 20:37:42.023111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:42.023079 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t4xvb"] Nov 03 20:37:42.181855 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:42.181819 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d3f5fd-69f3-4a98-8343-49a6dd2f3fef" path="/var/lib/kubelet/pods/52d3f5fd-69f3-4a98-8343-49a6dd2f3fef/volumes" Nov 03 20:37:56.112511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.112472 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf"] Nov 03 20:37:56.112958 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.112934 5111 cpu_manager.go:414] "containerMap: removing container" podUID="52d3f5fd-69f3-4a98-8343-49a6dd2f3fef" containerName="revoke-keys" Nov 03 20:37:56.112958 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.112947 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d3f5fd-69f3-4a98-8343-49a6dd2f3fef" containerName="revoke-keys" Nov 03 20:37:56.113034 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.113015 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d3f5fd-69f3-4a98-8343-49a6dd2f3fef" containerName="revoke-keys" Nov 03 20:37:56.117792 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.117773 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf" Nov 03 20:37:56.127020 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.126999 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:37:56.127139 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.127043 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:37:56.127139 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.127085 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:37:56.130724 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.130706 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:37:56.146516 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.146490 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf"] Nov 03 20:37:56.200888 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.200850 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f198916-f942-44dc-a871-88fe53e3bafb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8vbf\" (UID: \"3f198916-f942-44dc-a871-88fe53e3bafb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf" Nov 03 20:37:56.201087 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.200906 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zrhh\" (UniqueName: \"kubernetes.io/projected/3f198916-f942-44dc-a871-88fe53e3bafb-kube-api-access-9zrhh\") pod \"maas-api-revoke-sub-3d4156f1-v8vbf\" (UID: \"3f198916-f942-44dc-a871-88fe53e3bafb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf" Nov 03 20:37:56.201087 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.200985 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f198916-f942-44dc-a871-88fe53e3bafb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8vbf\" (UID: \"3f198916-f942-44dc-a871-88fe53e3bafb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf" Nov 03 20:37:56.302675 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.302629 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f198916-f942-44dc-a871-88fe53e3bafb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8vbf\" (UID: \"3f198916-f942-44dc-a871-88fe53e3bafb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf" Nov 03 20:37:56.302900 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.302843 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zrhh\" (UniqueName: \"kubernetes.io/projected/3f198916-f942-44dc-a871-88fe53e3bafb-kube-api-access-9zrhh\") pod \"maas-api-revoke-sub-3d4156f1-v8vbf\" (UID: \"3f198916-f942-44dc-a871-88fe53e3bafb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf" Nov 03 20:37:56.302983 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.302924 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f198916-f942-44dc-a871-88fe53e3bafb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8vbf\" (UID: \"3f198916-f942-44dc-a871-88fe53e3bafb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf" Nov 03 20:37:56.303446 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.303417 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f198916-f942-44dc-a871-88fe53e3bafb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8vbf\" (UID: \"3f198916-f942-44dc-a871-88fe53e3bafb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf" Nov 03 20:37:56.305410 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.305390 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f198916-f942-44dc-a871-88fe53e3bafb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8vbf\" (UID: \"3f198916-f942-44dc-a871-88fe53e3bafb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf" Nov 03 20:37:56.311403 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.311377 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zrhh\" (UniqueName: \"kubernetes.io/projected/3f198916-f942-44dc-a871-88fe53e3bafb-kube-api-access-9zrhh\") pod \"maas-api-revoke-sub-3d4156f1-v8vbf\" (UID: \"3f198916-f942-44dc-a871-88fe53e3bafb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf" Nov 03 20:37:56.430850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.430810 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf" Nov 03 20:37:56.708409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:56.707889 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf"] Nov 03 20:37:56.709203 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:37:56.709170 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f198916_f942_44dc_a871_88fe53e3bafb.slice/crio-b8c22330252a2966162ca5e01e01df89842d5ac8bbada6882aed2f2736968d98 WatchSource:0}: Error finding container b8c22330252a2966162ca5e01e01df89842d5ac8bbada6882aed2f2736968d98: Status 404 returned error can't find the container with id b8c22330252a2966162ca5e01e01df89842d5ac8bbada6882aed2f2736968d98 Nov 03 20:37:57.570825 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:57.570792 5111 generic.go:356] "Generic (PLEG): container finished" podID="3f198916-f942-44dc-a871-88fe53e3bafb" containerID="1aa570cf50f30c0d6f8e971f4065b3250d60cc2b371abd62249ffaebdf992fb3" exitCode=0 Nov 03 20:37:57.571318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:57.570879 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf" event={"ID":"3f198916-f942-44dc-a871-88fe53e3bafb","Type":"ContainerDied","Data":"1aa570cf50f30c0d6f8e971f4065b3250d60cc2b371abd62249ffaebdf992fb3"} Nov 03 20:37:57.571318 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:57.570940 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf" event={"ID":"3f198916-f942-44dc-a871-88fe53e3bafb","Type":"ContainerStarted","Data":"b8c22330252a2966162ca5e01e01df89842d5ac8bbada6882aed2f2736968d98"} Nov 03 20:37:58.717933 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:58.717904 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf" Nov 03 20:37:58.830772 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:58.830675 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f198916-f942-44dc-a871-88fe53e3bafb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f198916-f942-44dc-a871-88fe53e3bafb-maas-api-xks-ca\") pod \"3f198916-f942-44dc-a871-88fe53e3bafb\" (UID: \"3f198916-f942-44dc-a871-88fe53e3bafb\") " Nov 03 20:37:58.830965 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:58.830841 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f198916-f942-44dc-a871-88fe53e3bafb-kube-api-access-9zrhh\" (UniqueName: \"kubernetes.io/projected/3f198916-f942-44dc-a871-88fe53e3bafb-kube-api-access-9zrhh\") pod \"3f198916-f942-44dc-a871-88fe53e3bafb\" (UID: \"3f198916-f942-44dc-a871-88fe53e3bafb\") " Nov 03 20:37:58.830965 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:58.830953 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f198916-f942-44dc-a871-88fe53e3bafb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f198916-f942-44dc-a871-88fe53e3bafb-maas-api-openshift-ca\") pod \"3f198916-f942-44dc-a871-88fe53e3bafb\" (UID: \"3f198916-f942-44dc-a871-88fe53e3bafb\") " Nov 03 20:37:58.831299 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:58.831276 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f198916-f942-44dc-a871-88fe53e3bafb-maas-api-openshift-ca" pod "3f198916-f942-44dc-a871-88fe53e3bafb" (UID: "3f198916-f942-44dc-a871-88fe53e3bafb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:58.832840 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:58.832813 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f198916-f942-44dc-a871-88fe53e3bafb-maas-api-xks-ca" pod "3f198916-f942-44dc-a871-88fe53e3bafb" (UID: "3f198916-f942-44dc-a871-88fe53e3bafb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:58.833027 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:58.833003 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f198916-f942-44dc-a871-88fe53e3bafb-kube-api-access-9zrhh" pod "3f198916-f942-44dc-a871-88fe53e3bafb" (UID: "3f198916-f942-44dc-a871-88fe53e3bafb"). InnerVolumeSpecName "kube-api-access-9zrhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:58.931815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:58.931778 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zrhh\" (UniqueName: \"kubernetes.io/projected/3f198916-f942-44dc-a871-88fe53e3bafb-kube-api-access-9zrhh\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:58.931815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:58.931812 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f198916-f942-44dc-a871-88fe53e3bafb-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:58.931815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:58.931823 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f198916-f942-44dc-a871-88fe53e3bafb-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:59.582844 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:59.582810 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf" Nov 03 20:37:59.583127 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:59.582846 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf" event={"ID":"3f198916-f942-44dc-a871-88fe53e3bafb","Type":"ContainerDied","Data":"b8c22330252a2966162ca5e01e01df89842d5ac8bbada6882aed2f2736968d98"} Nov 03 20:37:59.583127 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:37:59.582900 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8c22330252a2966162ca5e01e01df89842d5ac8bbada6882aed2f2736968d98" Nov 03 20:38:03.403502 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:03.403455 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl"] Nov 03 20:38:03.404120 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:03.403729 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" podUID="cf6540f3-53cd-4088-bf2e-e44670ad75b5" containerName="istio-proxy" containerID="cri-o://17d12e28bc545d048b1502e48f9b99b07a62f71509428ee1a04a3e797e85bad9" gracePeriod=30 Nov 03 20:38:03.405783 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:03.405748 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx"] Nov 03 20:38:03.405958 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:03.405935 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" podUID="cf5856f8-f452-4f05-b083-fb88f3db028a" containerName="istio-proxy" containerID="cri-o://b34010d2adcefbfd7d4a1f28020eaf9d4c19f1f72b2a925cad89052bf186ef43" gracePeriod=30 Nov 03 20:38:05.057099 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:38:05.057057 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:38:08.413706 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.413663 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gmqdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:38:08.414264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.413731 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" podUID="cf6540f3-53cd-4088-bf2e-e44670ad75b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:38:08.438700 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.438662 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h7vtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:38:08.438868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.438724 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h7vtl" podUID="d969be31-5822-40ef-be6f-c6627bfa5956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:38:08.457699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.457658 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4vnbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:38:08.457893 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.457720 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" podUID="cf5856f8-f452-4f05-b083-fb88f3db028a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:38:08.632718 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.632684 5111 generic.go:356] "Generic (PLEG): container finished" podID="cf5856f8-f452-4f05-b083-fb88f3db028a" containerID="b34010d2adcefbfd7d4a1f28020eaf9d4c19f1f72b2a925cad89052bf186ef43" exitCode=0 Nov 03 20:38:08.632893 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.632764 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" event={"ID":"cf5856f8-f452-4f05-b083-fb88f3db028a","Type":"ContainerDied","Data":"b34010d2adcefbfd7d4a1f28020eaf9d4c19f1f72b2a925cad89052bf186ef43"} Nov 03 20:38:08.635642 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.635583 5111 generic.go:356] "Generic (PLEG): container finished" podID="cf6540f3-53cd-4088-bf2e-e44670ad75b5" containerID="17d12e28bc545d048b1502e48f9b99b07a62f71509428ee1a04a3e797e85bad9" exitCode=0 Nov 03 20:38:08.635903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.635843 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" event={"ID":"cf6540f3-53cd-4088-bf2e-e44670ad75b5","Type":"ContainerDied","Data":"17d12e28bc545d048b1502e48f9b99b07a62f71509428ee1a04a3e797e85bad9"} Nov 03 20:38:08.743140 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.743115 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 20:38:08.754590 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.750680 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 20:38:08.836989 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.836941 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-envoy\") pod \"cf5856f8-f452-4f05-b083-fb88f3db028a\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " Nov 03 20:38:08.837162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837017 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-podinfo\") pod \"cf5856f8-f452-4f05-b083-fb88f3db028a\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " Nov 03 20:38:08.837162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837054 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-workload-certs\") pod \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " Nov 03 20:38:08.837162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837095 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-credential-socket\") pod \"cf5856f8-f452-4f05-b083-fb88f3db028a\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " Nov 03 20:38:08.837162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837125 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-data\") pod \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " Nov 03 20:38:08.837162 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837161 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf5856f8-f452-4f05-b083-fb88f3db028a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf5856f8-f452-4f05-b083-fb88f3db028a-istiod-ca-cert\") pod \"cf5856f8-f452-4f05-b083-fb88f3db028a\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " Nov 03 20:38:08.837379 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837209 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-podinfo\") pod \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " Nov 03 20:38:08.837379 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837241 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-data\") pod \"cf5856f8-f452-4f05-b083-fb88f3db028a\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " Nov 03 20:38:08.837379 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837290 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6540f3-53cd-4088-bf2e-e44670ad75b5-kube-api-access-g2zfb\" (UniqueName: \"kubernetes.io/projected/cf6540f3-53cd-4088-bf2e-e44670ad75b5-kube-api-access-g2zfb\") pod \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " Nov 03 20:38:08.837379 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837317 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-credential-socket\") pod \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " Nov 03 20:38:08.837379 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837371 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istiod-ca-cert\") pod \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " Nov 03 20:38:08.837646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837399 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-envoy\") pod \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " Nov 03 20:38:08.837646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837415 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-data" pod "cf6540f3-53cd-4088-bf2e-e44670ad75b5" (UID: "cf6540f3-53cd-4088-bf2e-e44670ad75b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:08.837646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837435 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-workload-certs" pod "cf6540f3-53cd-4088-bf2e-e44670ad75b5" (UID: "cf6540f3-53cd-4088-bf2e-e44670ad75b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:08.837646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837462 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-token\" (UniqueName: \"kubernetes.io/projected/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-token\") pod \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " Nov 03 20:38:08.837646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837491 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-workload-socket\") pod \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\" (UID: \"cf6540f3-53cd-4088-bf2e-e44670ad75b5\") " Nov 03 20:38:08.837646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837569 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-token\" (UniqueName: \"kubernetes.io/projected/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-token\") pod \"cf5856f8-f452-4f05-b083-fb88f3db028a\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " Nov 03 20:38:08.837646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837611 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5856f8-f452-4f05-b083-fb88f3db028a-kube-api-access-6g2v6\" (UniqueName: \"kubernetes.io/projected/cf5856f8-f452-4f05-b083-fb88f3db028a-kube-api-access-6g2v6\") pod \"cf5856f8-f452-4f05-b083-fb88f3db028a\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " Nov 03 20:38:08.837646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837642 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-workload-socket\") pod \"cf5856f8-f452-4f05-b083-fb88f3db028a\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " Nov 03 20:38:08.837910 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.837691 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-workload-certs\") pod \"cf5856f8-f452-4f05-b083-fb88f3db028a\" (UID: \"cf5856f8-f452-4f05-b083-fb88f3db028a\") " Nov 03 20:38:08.838111 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.838087 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.838156 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.838119 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.838704 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.838582 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-credential-socket" pod "cf6540f3-53cd-4088-bf2e-e44670ad75b5" (UID: "cf6540f3-53cd-4088-bf2e-e44670ad75b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:08.839265 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.839203 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-credential-socket" pod "cf5856f8-f452-4f05-b083-fb88f3db028a" (UID: "cf5856f8-f452-4f05-b083-fb88f3db028a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:08.860029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.859926 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-data" pod "cf5856f8-f452-4f05-b083-fb88f3db028a" (UID: "cf5856f8-f452-4f05-b083-fb88f3db028a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:08.860029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.859964 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-envoy" pod "cf5856f8-f452-4f05-b083-fb88f3db028a" (UID: "cf5856f8-f452-4f05-b083-fb88f3db028a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:08.860563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.860512 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-podinfo" pod "cf5856f8-f452-4f05-b083-fb88f3db028a" (UID: "cf5856f8-f452-4f05-b083-fb88f3db028a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:38:08.860563 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.860553 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-podinfo" pod "cf6540f3-53cd-4088-bf2e-e44670ad75b5" (UID: "cf6540f3-53cd-4088-bf2e-e44670ad75b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:38:08.860768 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.860593 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5856f8-f452-4f05-b083-fb88f3db028a-istiod-ca-cert" pod "cf5856f8-f452-4f05-b083-fb88f3db028a" (UID: "cf5856f8-f452-4f05-b083-fb88f3db028a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:08.861155 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.860895 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-workload-socket" pod "cf6540f3-53cd-4088-bf2e-e44670ad75b5" (UID: "cf6540f3-53cd-4088-bf2e-e44670ad75b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:08.861155 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.861028 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-workload-socket" pod "cf5856f8-f452-4f05-b083-fb88f3db028a" (UID: "cf5856f8-f452-4f05-b083-fb88f3db028a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:08.861343 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.861324 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-workload-certs" pod "cf5856f8-f452-4f05-b083-fb88f3db028a" (UID: "cf5856f8-f452-4f05-b083-fb88f3db028a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:08.863385 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.863330 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6540f3-53cd-4088-bf2e-e44670ad75b5-kube-api-access-g2zfb" pod "cf6540f3-53cd-4088-bf2e-e44670ad75b5" (UID: "cf6540f3-53cd-4088-bf2e-e44670ad75b5"). InnerVolumeSpecName "kube-api-access-g2zfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:08.863486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.863425 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-envoy" pod "cf6540f3-53cd-4088-bf2e-e44670ad75b5" (UID: "cf6540f3-53cd-4088-bf2e-e44670ad75b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:08.864208 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.864179 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5856f8-f452-4f05-b083-fb88f3db028a-kube-api-access-6g2v6" pod "cf5856f8-f452-4f05-b083-fb88f3db028a" (UID: "cf5856f8-f452-4f05-b083-fb88f3db028a"). InnerVolumeSpecName "kube-api-access-6g2v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:08.864312 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.864240 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istiod-ca-cert" pod "cf6540f3-53cd-4088-bf2e-e44670ad75b5" (UID: "cf6540f3-53cd-4088-bf2e-e44670ad75b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:08.864694 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.864669 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-token" pod "cf5856f8-f452-4f05-b083-fb88f3db028a" (UID: "cf5856f8-f452-4f05-b083-fb88f3db028a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:08.865178 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.865150 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-token" pod "cf6540f3-53cd-4088-bf2e-e44670ad75b5" (UID: "cf6540f3-53cd-4088-bf2e-e44670ad75b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:08.939201 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.939158 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.939201 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.939195 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g2v6\" (UniqueName: \"kubernetes.io/projected/cf5856f8-f452-4f05-b083-fb88f3db028a-kube-api-access-6g2v6\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.939201 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.939208 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.939505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.939222 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.939505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.939235 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.939505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.939249 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.939505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.939260 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.939505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.939271 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf5856f8-f452-4f05-b083-fb88f3db028a-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.939505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.939286 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.939505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.939297 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf5856f8-f452-4f05-b083-fb88f3db028a-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.939505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.939312 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2zfb\" (UniqueName: \"kubernetes.io/projected/cf6540f3-53cd-4088-bf2e-e44670ad75b5-kube-api-access-g2zfb\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.939505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.939324 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.939505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.939336 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.939505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.939348 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.939505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.939361 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6540f3-53cd-4088-bf2e-e44670ad75b5-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.939505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:08.939373 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6540f3-53cd-4088-bf2e-e44670ad75b5-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:09.642031 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:09.641994 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" Nov 03 20:38:09.642582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:09.641999 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl" event={"ID":"cf6540f3-53cd-4088-bf2e-e44670ad75b5","Type":"ContainerDied","Data":"92476df310503b1a5a2db2fff65d9f114c1d355c3829747ae2e7b64456988b32"} Nov 03 20:38:09.642582 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:09.642148 5111 scope.go:122] "RemoveContainer" containerID="17d12e28bc545d048b1502e48f9b99b07a62f71509428ee1a04a3e797e85bad9" Nov 03 20:38:09.643588 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:09.643562 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" event={"ID":"cf5856f8-f452-4f05-b083-fb88f3db028a","Type":"ContainerDied","Data":"b2745530e338da77b8b2b3d0b280d5ecfd8eeb3afa0db05df782c64e70ff7731"} Nov 03 20:38:09.643681 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:09.643597 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx" Nov 03 20:38:09.652687 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:09.652663 5111 scope.go:122] "RemoveContainer" containerID="b34010d2adcefbfd7d4a1f28020eaf9d4c19f1f72b2a925cad89052bf186ef43" Nov 03 20:38:09.668319 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:09.668292 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl"] Nov 03 20:38:09.672009 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:09.671987 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gmqdl"] Nov 03 20:38:09.683194 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:09.683127 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx"] Nov 03 20:38:09.689425 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:09.689396 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4vnbx"] Nov 03 20:38:10.181649 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:10.181617 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf5856f8-f452-4f05-b083-fb88f3db028a" path="/var/lib/kubelet/pods/cf5856f8-f452-4f05-b083-fb88f3db028a/volumes" Nov 03 20:38:10.182876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:10.182830 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf6540f3-53cd-4088-bf2e-e44670ad75b5" path="/var/lib/kubelet/pods/cf6540f3-53cd-4088-bf2e-e44670ad75b5/volumes" Nov 03 20:38:18.451269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.451232 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd"] Nov 03 20:38:18.451893 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.451871 5111 cpu_manager.go:414] "containerMap: removing container" podUID="cf5856f8-f452-4f05-b083-fb88f3db028a" containerName="istio-proxy" Nov 03 20:38:18.451957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.451897 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5856f8-f452-4f05-b083-fb88f3db028a" containerName="istio-proxy" Nov 03 20:38:18.451957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.451935 5111 cpu_manager.go:414] "containerMap: removing container" podUID="cf6540f3-53cd-4088-bf2e-e44670ad75b5" containerName="istio-proxy" Nov 03 20:38:18.451957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.451944 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf6540f3-53cd-4088-bf2e-e44670ad75b5" containerName="istio-proxy" Nov 03 20:38:18.452086 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.451959 5111 cpu_manager.go:414] "containerMap: removing container" podUID="3f198916-f942-44dc-a871-88fe53e3bafb" containerName="revoke-keys" Nov 03 20:38:18.452086 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.451967 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f198916-f942-44dc-a871-88fe53e3bafb" containerName="revoke-keys" Nov 03 20:38:18.452086 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.452082 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf6540f3-53cd-4088-bf2e-e44670ad75b5" containerName="istio-proxy" Nov 03 20:38:18.452186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.452099 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f198916-f942-44dc-a871-88fe53e3bafb" containerName="revoke-keys" Nov 03 20:38:18.452186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.452114 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5856f8-f452-4f05-b083-fb88f3db028a" containerName="istio-proxy" Nov 03 20:38:18.457022 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.457003 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.465303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.465230 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd"] Nov 03 20:38:18.537252 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.537216 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd67r\" (UniqueName: \"kubernetes.io/projected/ea1836fd-1444-44ab-831c-165ef7470a5c-kube-api-access-zd67r\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.537252 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.537256 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea1836fd-1444-44ab-831c-165ef7470a5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.537467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.537281 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea1836fd-1444-44ab-831c-165ef7470a5c-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.537467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.537377 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea1836fd-1444-44ab-831c-165ef7470a5c-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.537467 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.537428 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1836fd-1444-44ab-831c-165ef7470a5c-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.537611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.537498 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1836fd-1444-44ab-831c-165ef7470a5c-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.537611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.537525 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea1836fd-1444-44ab-831c-165ef7470a5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.537611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.537607 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea1836fd-1444-44ab-831c-165ef7470a5c-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.537715 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.537639 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea1836fd-1444-44ab-831c-165ef7470a5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.638402 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.638361 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1836fd-1444-44ab-831c-165ef7470a5c-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.638657 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.638408 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1836fd-1444-44ab-831c-165ef7470a5c-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.638657 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.638440 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea1836fd-1444-44ab-831c-165ef7470a5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.638657 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.638478 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea1836fd-1444-44ab-831c-165ef7470a5c-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.638657 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.638507 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea1836fd-1444-44ab-831c-165ef7470a5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.638657 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.638585 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd67r\" (UniqueName: \"kubernetes.io/projected/ea1836fd-1444-44ab-831c-165ef7470a5c-kube-api-access-zd67r\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.638657 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.638610 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea1836fd-1444-44ab-831c-165ef7470a5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.638657 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.638637 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea1836fd-1444-44ab-831c-165ef7470a5c-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.639079 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.638685 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea1836fd-1444-44ab-831c-165ef7470a5c-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.639079 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.638877 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1836fd-1444-44ab-831c-165ef7470a5c-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.639186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.639095 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea1836fd-1444-44ab-831c-165ef7470a5c-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.639186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.639113 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1836fd-1444-44ab-831c-165ef7470a5c-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.639186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.639136 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea1836fd-1444-44ab-831c-165ef7470a5c-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.639478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.639458 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea1836fd-1444-44ab-831c-165ef7470a5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.641023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.641001 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea1836fd-1444-44ab-831c-165ef7470a5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.641268 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.641252 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea1836fd-1444-44ab-831c-165ef7470a5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.647322 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.647292 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea1836fd-1444-44ab-831c-165ef7470a5c-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.647594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.647516 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd67r\" (UniqueName: \"kubernetes.io/projected/ea1836fd-1444-44ab-831c-165ef7470a5c-kube-api-access-zd67r\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8bcwd\" (UID: \"ea1836fd-1444-44ab-831c-165ef7470a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.775880 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.775786 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:18.850809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.848392 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw"] Nov 03 20:38:18.854364 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.854318 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:38:18.871603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.865306 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7bzpm\"" Nov 03 20:38:18.871603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.867720 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-4442a4-serving-cert\"" Nov 03 20:38:18.871603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.867767 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:38:18.871603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.867855 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:38:18.871603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.867972 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:38:18.871603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.869650 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw"] Nov 03 20:38:18.904621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.904584 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn"] Nov 03 20:38:18.908269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.908250 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" Nov 03 20:38:18.919108 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.919036 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn"] Nov 03 20:38:18.941981 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.941941 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa034437-95d0-47ad-ae39-767115dffec9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw\" (UID: \"fa034437-95d0-47ad-ae39-767115dffec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:38:18.942180 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.942007 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa034437-95d0-47ad-ae39-767115dffec9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw\" (UID: \"fa034437-95d0-47ad-ae39-767115dffec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:38:18.942180 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.942116 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk7cf\" (UniqueName: \"kubernetes.io/projected/fa034437-95d0-47ad-ae39-767115dffec9-kube-api-access-dk7cf\") pod \"maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw\" (UID: \"fa034437-95d0-47ad-ae39-767115dffec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:38:18.942297 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.942197 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6168df8b-cb4f-4e2f-87f4-8a1138ee32ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn\" (UID: \"6168df8b-cb4f-4e2f-87f4-8a1138ee32ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" Nov 03 20:38:18.942345 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.942319 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa034437-95d0-47ad-ae39-767115dffec9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw\" (UID: \"fa034437-95d0-47ad-ae39-767115dffec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:38:18.942395 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.942369 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmbcr\" (UniqueName: \"kubernetes.io/projected/6168df8b-cb4f-4e2f-87f4-8a1138ee32ff-kube-api-access-zmbcr\") pod \"payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn\" (UID: \"6168df8b-cb4f-4e2f-87f4-8a1138ee32ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" Nov 03 20:38:18.960608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.960576 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4442a4\"" Nov 03 20:38:18.960806 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.960674 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bc323086-dockercfg-n8h9v\"" Nov 03 20:38:18.971256 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.971228 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d"] Nov 03 20:38:18.975435 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.975414 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" Nov 03 20:38:18.984699 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:18.984612 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d"] Nov 03 20:38:19.044071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.043989 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmbcr\" (UniqueName: \"kubernetes.io/projected/6168df8b-cb4f-4e2f-87f4-8a1138ee32ff-kube-api-access-zmbcr\") pod \"payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn\" (UID: \"6168df8b-cb4f-4e2f-87f4-8a1138ee32ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" Nov 03 20:38:19.044071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.044038 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ndst\" (UniqueName: \"kubernetes.io/projected/658bb868-b206-4256-806e-10eb7f091599-kube-api-access-7ndst\") pod \"payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d\" (UID: \"658bb868-b206-4256-806e-10eb7f091599\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" Nov 03 20:38:19.044071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.044071 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa034437-95d0-47ad-ae39-767115dffec9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw\" (UID: \"fa034437-95d0-47ad-ae39-767115dffec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:38:19.044303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.044099 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa034437-95d0-47ad-ae39-767115dffec9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw\" (UID: \"fa034437-95d0-47ad-ae39-767115dffec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:38:19.044303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.044125 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk7cf\" (UniqueName: \"kubernetes.io/projected/fa034437-95d0-47ad-ae39-767115dffec9-kube-api-access-dk7cf\") pod \"maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw\" (UID: \"fa034437-95d0-47ad-ae39-767115dffec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:38:19.044303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.044193 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/658bb868-b206-4256-806e-10eb7f091599-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d\" (UID: \"658bb868-b206-4256-806e-10eb7f091599\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" Nov 03 20:38:19.044303 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.044282 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6168df8b-cb4f-4e2f-87f4-8a1138ee32ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn\" (UID: \"6168df8b-cb4f-4e2f-87f4-8a1138ee32ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" Nov 03 20:38:19.044447 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.044304 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa034437-95d0-47ad-ae39-767115dffec9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw\" (UID: \"fa034437-95d0-47ad-ae39-767115dffec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:38:19.044987 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.044961 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa034437-95d0-47ad-ae39-767115dffec9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw\" (UID: \"fa034437-95d0-47ad-ae39-767115dffec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:38:19.045131 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.045091 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6168df8b-cb4f-4e2f-87f4-8a1138ee32ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn\" (UID: \"6168df8b-cb4f-4e2f-87f4-8a1138ee32ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" Nov 03 20:38:19.047106 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.047086 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa034437-95d0-47ad-ae39-767115dffec9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw\" (UID: \"fa034437-95d0-47ad-ae39-767115dffec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:38:19.053425 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.053210 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa034437-95d0-47ad-ae39-767115dffec9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw\" (UID: \"fa034437-95d0-47ad-ae39-767115dffec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:38:19.053425 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.053262 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmbcr\" (UniqueName: \"kubernetes.io/projected/6168df8b-cb4f-4e2f-87f4-8a1138ee32ff-kube-api-access-zmbcr\") pod \"payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn\" (UID: \"6168df8b-cb4f-4e2f-87f4-8a1138ee32ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" Nov 03 20:38:19.055361 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.055335 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk7cf\" (UniqueName: \"kubernetes.io/projected/fa034437-95d0-47ad-ae39-767115dffec9-kube-api-access-dk7cf\") pod \"maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw\" (UID: \"fa034437-95d0-47ad-ae39-767115dffec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:38:19.074687 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.074645 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd"] Nov 03 20:38:19.077065 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:38:19.077033 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea1836fd_1444_44ab_831c_165ef7470a5c.slice/crio-d00054adff669668e27921218df4b24c7697dd5f5dddb017b90c4a00a36a9ab4 WatchSource:0}: Error finding container d00054adff669668e27921218df4b24c7697dd5f5dddb017b90c4a00a36a9ab4: Status 404 returned error can't find the container with id d00054adff669668e27921218df4b24c7697dd5f5dddb017b90c4a00a36a9ab4 Nov 03 20:38:19.079389 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.079356 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:38:19.079505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.079423 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:38:19.079505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.079452 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:38:19.145840 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.145808 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ndst\" (UniqueName: \"kubernetes.io/projected/658bb868-b206-4256-806e-10eb7f091599-kube-api-access-7ndst\") pod \"payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d\" (UID: \"658bb868-b206-4256-806e-10eb7f091599\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" Nov 03 20:38:19.145982 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.145964 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/658bb868-b206-4256-806e-10eb7f091599-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d\" (UID: \"658bb868-b206-4256-806e-10eb7f091599\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" Nov 03 20:38:19.146717 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.146698 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/658bb868-b206-4256-806e-10eb7f091599-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d\" (UID: \"658bb868-b206-4256-806e-10eb7f091599\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" Nov 03 20:38:19.153969 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.153946 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ndst\" (UniqueName: \"kubernetes.io/projected/658bb868-b206-4256-806e-10eb7f091599-kube-api-access-7ndst\") pod \"payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d\" (UID: \"658bb868-b206-4256-806e-10eb7f091599\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" Nov 03 20:38:19.181365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.181330 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:38:19.234787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.233999 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" Nov 03 20:38:19.291113 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.291068 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" Nov 03 20:38:19.461082 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.461049 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw"] Nov 03 20:38:19.463694 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:38:19.463025 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa034437_95d0_47ad_ae39_767115dffec9.slice/crio-4b238fb0fe24a0507e568b8bcda8ea57903c66059226802eb25c0100eb97fa2c WatchSource:0}: Error finding container 4b238fb0fe24a0507e568b8bcda8ea57903c66059226802eb25c0100eb97fa2c: Status 404 returned error can't find the container with id 4b238fb0fe24a0507e568b8bcda8ea57903c66059226802eb25c0100eb97fa2c Nov 03 20:38:19.466179 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.466155 5111 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:38:19.531285 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.531239 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn"] Nov 03 20:38:19.593133 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.593111 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d"] Nov 03 20:38:19.601826 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:38:19.601796 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658bb868_b206_4256_806e_10eb7f091599.slice/crio-85f0b10680517b8c92b5be194371b2c10020cb93cccb71a7abb902a5381d88c5 WatchSource:0}: Error finding container 85f0b10680517b8c92b5be194371b2c10020cb93cccb71a7abb902a5381d88c5: Status 404 returned error can't find the container with id 85f0b10680517b8c92b5be194371b2c10020cb93cccb71a7abb902a5381d88c5 Nov 03 20:38:19.697544 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.697504 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" event={"ID":"fa034437-95d0-47ad-ae39-767115dffec9","Type":"ContainerStarted","Data":"4b238fb0fe24a0507e568b8bcda8ea57903c66059226802eb25c0100eb97fa2c"} Nov 03 20:38:19.699081 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.699047 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" event={"ID":"6168df8b-cb4f-4e2f-87f4-8a1138ee32ff","Type":"ContainerStarted","Data":"b3fb31ea3692c009ad198124779ecd4abdf10e77a35bc995ae8e0682ba7fff6b"} Nov 03 20:38:19.699238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.699105 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" Nov 03 20:38:19.699238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.699128 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" event={"ID":"6168df8b-cb4f-4e2f-87f4-8a1138ee32ff","Type":"ContainerStarted","Data":"a94e2fdd8a82f06c7aa2c64633d30e686af50936a7c316c81cbcc8793498c36f"} Nov 03 20:38:19.700611 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.700582 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" event={"ID":"ea1836fd-1444-44ab-831c-165ef7470a5c","Type":"ContainerStarted","Data":"829f0bd61d752c1054b12757e1727c1ccd7b967a162fd236ac148318e52d0996"} Nov 03 20:38:19.700732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.700628 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" event={"ID":"ea1836fd-1444-44ab-831c-165ef7470a5c","Type":"ContainerStarted","Data":"d00054adff669668e27921218df4b24c7697dd5f5dddb017b90c4a00a36a9ab4"} Nov 03 20:38:19.702052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.702030 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" event={"ID":"658bb868-b206-4256-806e-10eb7f091599","Type":"ContainerStarted","Data":"59506ee563f53d76a63fc1d19295cddc76a5faa045a3ba71d8b58a15337baf3d"} Nov 03 20:38:19.702156 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.702073 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" event={"ID":"658bb868-b206-4256-806e-10eb7f091599","Type":"ContainerStarted","Data":"85f0b10680517b8c92b5be194371b2c10020cb93cccb71a7abb902a5381d88c5"} Nov 03 20:38:19.702156 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.702119 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" Nov 03 20:38:19.719907 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.719865 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" podStartSLOduration=1.719852554 podStartE2EDuration="1.719852554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:19.716490971 +0000 UTC m=+3529.678869409" watchObservedRunningTime="2026-11-03 20:38:19.719852554 +0000 UTC m=+3529.682230993" Nov 03 20:38:19.738478 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.737645 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" podStartSLOduration=1.737628885 podStartE2EDuration="1.737628885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:19.737199404 +0000 UTC m=+3529.699577843" watchObservedRunningTime="2026-11-03 20:38:19.737628885 +0000 UTC m=+3529.700007325" Nov 03 20:38:19.768294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.768228 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" podStartSLOduration=1.768208752 podStartE2EDuration="1.768208752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:19.766013479 +0000 UTC m=+3529.728391920" watchObservedRunningTime="2026-11-03 20:38:19.768208752 +0000 UTC m=+3529.730587192" Nov 03 20:38:19.775898 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:19.775860 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:20.709524 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:20.709479 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" event={"ID":"fa034437-95d0-47ad-ae39-767115dffec9","Type":"ContainerStarted","Data":"60761ec6307e6d19d792a520e375181180f7f6eaa166b59e061fc55912eb9580"} Nov 03 20:38:20.710055 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:20.710033 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:38:20.730363 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:20.730300 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" podStartSLOduration=2.189103859 podStartE2EDuration="2.730281559s" totalImagesPullingTime="541.1777ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:20.728069118 +0000 UTC m=+3530.690447581" watchObservedRunningTime="2026-11-03 20:38:20.730281559 +0000 UTC m=+3530.692660000" Nov 03 20:38:20.778466 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:20.778429 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8bcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 20:38:20.778690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:20.778488 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" podUID="ea1836fd-1444-44ab-831c-165ef7470a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 20:38:21.776441 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:21.776401 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8bcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 20:38:21.776907 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:21.776470 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" podUID="ea1836fd-1444-44ab-831c-165ef7470a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 20:38:22.776511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:22.776468 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8bcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 20:38:22.777040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:22.776552 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" podUID="ea1836fd-1444-44ab-831c-165ef7470a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 20:38:23.776231 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:23.776184 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8bcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 20:38:23.776462 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:23.776268 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" podUID="ea1836fd-1444-44ab-831c-165ef7470a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 20:38:24.776474 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:24.776431 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8bcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 20:38:24.776949 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:24.776521 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" podUID="ea1836fd-1444-44ab-831c-165ef7470a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 20:38:25.776473 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:25.776427 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8bcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 20:38:25.776692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:25.776524 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" podUID="ea1836fd-1444-44ab-831c-165ef7470a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 20:38:27.439675 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:27.439644 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:27.440184 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:27.440167 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:27.440954 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:27.440930 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" Nov 03 20:38:27.724882 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:27.724801 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:38:30.713030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:30.713000 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" Nov 03 20:38:30.713595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:30.713067 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" Nov 03 20:38:35.057523 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:38:35.057484 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:38:39.112370 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:38:39.112330 5111 scope.go:122] "RemoveContainer" containerID="a72f8982bb196703e5ad67679909b78666411908d1eaf96f7d6a3a6d14ecfe7f" Nov 03 20:39:05.057054 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:39:05.057008 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:39:25.711429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:25.711382 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d"] Nov 03 20:39:25.725216 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:25.725182 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn"] Nov 03 20:39:26.850019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:26.849985 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw"] Nov 03 20:39:26.850507 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:26.850210 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" podUID="fa034437-95d0-47ad-ae39-767115dffec9" containerName="maas-api" containerID="cri-o://60761ec6307e6d19d792a520e375181180f7f6eaa166b59e061fc55912eb9580" gracePeriod=30 Nov 03 20:39:27.306523 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:27.306493 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:39:27.481430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:27.481349 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa034437-95d0-47ad-ae39-767115dffec9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa034437-95d0-47ad-ae39-767115dffec9-metrics-certs\") pod \"fa034437-95d0-47ad-ae39-767115dffec9\" (UID: \"fa034437-95d0-47ad-ae39-767115dffec9\") " Nov 03 20:39:27.481430 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:27.481404 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa034437-95d0-47ad-ae39-767115dffec9-kube-api-access-dk7cf\" (UniqueName: \"kubernetes.io/projected/fa034437-95d0-47ad-ae39-767115dffec9-kube-api-access-dk7cf\") pod \"fa034437-95d0-47ad-ae39-767115dffec9\" (UID: \"fa034437-95d0-47ad-ae39-767115dffec9\") " Nov 03 20:39:27.481731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:27.481459 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa034437-95d0-47ad-ae39-767115dffec9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa034437-95d0-47ad-ae39-767115dffec9-openshift-service-ca\") pod \"fa034437-95d0-47ad-ae39-767115dffec9\" (UID: \"fa034437-95d0-47ad-ae39-767115dffec9\") " Nov 03 20:39:27.481731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:27.481587 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa034437-95d0-47ad-ae39-767115dffec9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa034437-95d0-47ad-ae39-767115dffec9-maas-api-tls\") pod \"fa034437-95d0-47ad-ae39-767115dffec9\" (UID: \"fa034437-95d0-47ad-ae39-767115dffec9\") " Nov 03 20:39:27.481860 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:27.481839 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa034437-95d0-47ad-ae39-767115dffec9-openshift-service-ca" pod "fa034437-95d0-47ad-ae39-767115dffec9" (UID: "fa034437-95d0-47ad-ae39-767115dffec9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:27.483694 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:27.483671 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa034437-95d0-47ad-ae39-767115dffec9-metrics-certs" pod "fa034437-95d0-47ad-ae39-767115dffec9" (UID: "fa034437-95d0-47ad-ae39-767115dffec9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:27.483778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:27.483690 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa034437-95d0-47ad-ae39-767115dffec9-maas-api-tls" pod "fa034437-95d0-47ad-ae39-767115dffec9" (UID: "fa034437-95d0-47ad-ae39-767115dffec9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:27.483778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:27.483724 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa034437-95d0-47ad-ae39-767115dffec9-kube-api-access-dk7cf" pod "fa034437-95d0-47ad-ae39-767115dffec9" (UID: "fa034437-95d0-47ad-ae39-767115dffec9"). InnerVolumeSpecName "kube-api-access-dk7cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:27.583264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:27.583225 5111 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa034437-95d0-47ad-ae39-767115dffec9-openshift-service-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:27.583264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:27.583259 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa034437-95d0-47ad-ae39-767115dffec9-maas-api-tls\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:27.583264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:27.583269 5111 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa034437-95d0-47ad-ae39-767115dffec9-metrics-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:27.583522 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:27.583279 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk7cf\" (UniqueName: \"kubernetes.io/projected/fa034437-95d0-47ad-ae39-767115dffec9-kube-api-access-dk7cf\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:28.075136 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:28.075099 5111 generic.go:356] "Generic (PLEG): container finished" podID="fa034437-95d0-47ad-ae39-767115dffec9" containerID="60761ec6307e6d19d792a520e375181180f7f6eaa166b59e061fc55912eb9580" exitCode=0 Nov 03 20:39:28.075616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:28.075146 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" Nov 03 20:39:28.075616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:28.075187 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" event={"ID":"fa034437-95d0-47ad-ae39-767115dffec9","Type":"ContainerDied","Data":"60761ec6307e6d19d792a520e375181180f7f6eaa166b59e061fc55912eb9580"} Nov 03 20:39:28.075616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:28.075268 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw" event={"ID":"fa034437-95d0-47ad-ae39-767115dffec9","Type":"ContainerDied","Data":"4b238fb0fe24a0507e568b8bcda8ea57903c66059226802eb25c0100eb97fa2c"} Nov 03 20:39:28.075616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:28.075293 5111 scope.go:122] "RemoveContainer" containerID="60761ec6307e6d19d792a520e375181180f7f6eaa166b59e061fc55912eb9580" Nov 03 20:39:28.088770 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:28.088749 5111 scope.go:122] "RemoveContainer" containerID="60761ec6307e6d19d792a520e375181180f7f6eaa166b59e061fc55912eb9580" Nov 03 20:39:28.089080 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:39:28.089057 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60761ec6307e6d19d792a520e375181180f7f6eaa166b59e061fc55912eb9580\": container with ID starting with 60761ec6307e6d19d792a520e375181180f7f6eaa166b59e061fc55912eb9580 not found: ID does not exist" containerID="60761ec6307e6d19d792a520e375181180f7f6eaa166b59e061fc55912eb9580" Nov 03 20:39:28.089171 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:28.089093 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60761ec6307e6d19d792a520e375181180f7f6eaa166b59e061fc55912eb9580"} err="failed to get container status \"60761ec6307e6d19d792a520e375181180f7f6eaa166b59e061fc55912eb9580\": rpc error: code = NotFound desc = could not find container \"60761ec6307e6d19d792a520e375181180f7f6eaa166b59e061fc55912eb9580\": container with ID starting with 60761ec6307e6d19d792a520e375181180f7f6eaa166b59e061fc55912eb9580 not found: ID does not exist" Nov 03 20:39:28.102167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:28.102142 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw"] Nov 03 20:39:28.108445 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:28.108423 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4442a4-5596bc7cd9-thlmw"] Nov 03 20:39:28.182448 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:28.182414 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa034437-95d0-47ad-ae39-767115dffec9" path="/var/lib/kubelet/pods/fa034437-95d0-47ad-ae39-767115dffec9/volumes" Nov 03 20:39:30.712198 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:30.712163 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" podUID="658bb868-b206-4256-806e-10eb7f091599" containerName="payload-processing" containerID="cri-o://59506ee563f53d76a63fc1d19295cddc76a5faa045a3ba71d8b58a15337baf3d" gracePeriod=25 Nov 03 20:39:30.725929 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:30.725885 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" podUID="6168df8b-cb4f-4e2f-87f4-8a1138ee32ff" containerName="payload-pre-processing" containerID="cri-o://b3fb31ea3692c009ad198124779ecd4abdf10e77a35bc995ae8e0682ba7fff6b" gracePeriod=25 Nov 03 20:39:31.010610 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:31.010500 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:39:31.013551 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:31.013505 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:39:31.015238 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:31.015215 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:39:31.018615 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:31.018598 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:39:34.848711 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:34.848673 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf"] Nov 03 20:39:34.854059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:34.854032 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vbf"] Nov 03 20:39:34.894698 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:34.894667 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t"] Nov 03 20:39:34.895197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:34.895177 5111 cpu_manager.go:414] "containerMap: removing container" podUID="fa034437-95d0-47ad-ae39-767115dffec9" containerName="maas-api" Nov 03 20:39:34.895296 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:34.895199 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa034437-95d0-47ad-ae39-767115dffec9" containerName="maas-api" Nov 03 20:39:34.895363 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:34.895348 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa034437-95d0-47ad-ae39-767115dffec9" containerName="maas-api" Nov 03 20:39:34.900096 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:34.900074 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" Nov 03 20:39:34.904654 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:34.904631 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:39:34.904818 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:34.904678 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:39:34.904818 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:34.904813 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:39:34.904967 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:34.904818 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:39:34.905599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:34.905576 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t"] Nov 03 20:39:34.953094 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:34.953058 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6579d9f-39e4-46e0-935c-333653d0615b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tj64t\" (UID: \"b6579d9f-39e4-46e0-935c-333653d0615b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" Nov 03 20:39:34.953266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:34.953171 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbhnt\" (UniqueName: \"kubernetes.io/projected/b6579d9f-39e4-46e0-935c-333653d0615b-kube-api-access-qbhnt\") pod \"maas-api-revoke-sub-3d4156f1-tj64t\" (UID: \"b6579d9f-39e4-46e0-935c-333653d0615b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" Nov 03 20:39:34.953266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:34.953199 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6579d9f-39e4-46e0-935c-333653d0615b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tj64t\" (UID: \"b6579d9f-39e4-46e0-935c-333653d0615b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" Nov 03 20:39:35.054391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:35.054357 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6579d9f-39e4-46e0-935c-333653d0615b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tj64t\" (UID: \"b6579d9f-39e4-46e0-935c-333653d0615b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" Nov 03 20:39:35.054603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:35.054436 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhnt\" (UniqueName: \"kubernetes.io/projected/b6579d9f-39e4-46e0-935c-333653d0615b-kube-api-access-qbhnt\") pod \"maas-api-revoke-sub-3d4156f1-tj64t\" (UID: \"b6579d9f-39e4-46e0-935c-333653d0615b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" Nov 03 20:39:35.054603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:35.054458 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6579d9f-39e4-46e0-935c-333653d0615b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tj64t\" (UID: \"b6579d9f-39e4-46e0-935c-333653d0615b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" Nov 03 20:39:35.055313 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:35.055273 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6579d9f-39e4-46e0-935c-333653d0615b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tj64t\" (UID: \"b6579d9f-39e4-46e0-935c-333653d0615b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" Nov 03 20:39:35.057144 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:35.057117 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6579d9f-39e4-46e0-935c-333653d0615b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tj64t\" (UID: \"b6579d9f-39e4-46e0-935c-333653d0615b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" Nov 03 20:39:35.062019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:35.061993 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhnt\" (UniqueName: \"kubernetes.io/projected/b6579d9f-39e4-46e0-935c-333653d0615b-kube-api-access-qbhnt\") pod \"maas-api-revoke-sub-3d4156f1-tj64t\" (UID: \"b6579d9f-39e4-46e0-935c-333653d0615b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" Nov 03 20:39:35.070249 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:39:35.070219 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:39:35.215177 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:35.215142 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" Nov 03 20:39:35.487209 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:35.487156 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t"] Nov 03 20:39:35.489983 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:39:35.489954 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6579d9f_39e4_46e0_935c_333653d0615b.slice/crio-284b192f5e3e9abaa5bc6940f430749934e95c537e8aa4910756a0b0d8c2c95d WatchSource:0}: Error finding container 284b192f5e3e9abaa5bc6940f430749934e95c537e8aa4910756a0b0d8c2c95d: Status 404 returned error can't find the container with id 284b192f5e3e9abaa5bc6940f430749934e95c537e8aa4910756a0b0d8c2c95d Nov 03 20:39:36.118929 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:36.118899 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" event={"ID":"b6579d9f-39e4-46e0-935c-333653d0615b","Type":"ContainerStarted","Data":"58c24c1fa05451cb579a89bdaba6523b9f1e067efc736db55e930c0b94fc1650"} Nov 03 20:39:36.119421 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:36.118961 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" event={"ID":"b6579d9f-39e4-46e0-935c-333653d0615b","Type":"ContainerStarted","Data":"284b192f5e3e9abaa5bc6940f430749934e95c537e8aa4910756a0b0d8c2c95d"} Nov 03 20:39:36.136368 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:36.136312 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" podStartSLOduration=2.1363002030000002 podStartE2EDuration="2.136300203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:36.134473253 +0000 UTC m=+3606.096851692" watchObservedRunningTime="2026-11-03 20:39:36.136300203 +0000 UTC m=+3606.098678642" Nov 03 20:39:36.181957 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:36.181921 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f198916-f942-44dc-a871-88fe53e3bafb" path="/var/lib/kubelet/pods/3f198916-f942-44dc-a871-88fe53e3bafb/volumes" Nov 03 20:39:36.384912 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:39:36.384821 5111 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6579d9f_39e4_46e0_935c_333653d0615b.slice/crio-58c24c1fa05451cb579a89bdaba6523b9f1e067efc736db55e930c0b94fc1650.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6579d9f_39e4_46e0_935c_333653d0615b.slice/crio-conmon-58c24c1fa05451cb579a89bdaba6523b9f1e067efc736db55e930c0b94fc1650.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:39:37.125326 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:37.125291 5111 generic.go:356] "Generic (PLEG): container finished" podID="b6579d9f-39e4-46e0-935c-333653d0615b" containerID="58c24c1fa05451cb579a89bdaba6523b9f1e067efc736db55e930c0b94fc1650" exitCode=0 Nov 03 20:39:37.125810 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:37.125379 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" event={"ID":"b6579d9f-39e4-46e0-935c-333653d0615b","Type":"ContainerDied","Data":"58c24c1fa05451cb579a89bdaba6523b9f1e067efc736db55e930c0b94fc1650"} Nov 03 20:39:38.278078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:38.278051 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" Nov 03 20:39:38.394597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:38.394484 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6579d9f-39e4-46e0-935c-333653d0615b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6579d9f-39e4-46e0-935c-333653d0615b-maas-api-openshift-ca\") pod \"b6579d9f-39e4-46e0-935c-333653d0615b\" (UID: \"b6579d9f-39e4-46e0-935c-333653d0615b\") " Nov 03 20:39:38.394597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:38.394580 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6579d9f-39e4-46e0-935c-333653d0615b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6579d9f-39e4-46e0-935c-333653d0615b-maas-api-xks-ca\") pod \"b6579d9f-39e4-46e0-935c-333653d0615b\" (UID: \"b6579d9f-39e4-46e0-935c-333653d0615b\") " Nov 03 20:39:38.394844 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:38.394615 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6579d9f-39e4-46e0-935c-333653d0615b-kube-api-access-qbhnt\" (UniqueName: \"kubernetes.io/projected/b6579d9f-39e4-46e0-935c-333653d0615b-kube-api-access-qbhnt\") pod \"b6579d9f-39e4-46e0-935c-333653d0615b\" (UID: \"b6579d9f-39e4-46e0-935c-333653d0615b\") " Nov 03 20:39:38.394920 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:38.394868 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6579d9f-39e4-46e0-935c-333653d0615b-maas-api-openshift-ca" pod "b6579d9f-39e4-46e0-935c-333653d0615b" (UID: "b6579d9f-39e4-46e0-935c-333653d0615b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:38.396696 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:38.396675 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6579d9f-39e4-46e0-935c-333653d0615b-maas-api-xks-ca" pod "b6579d9f-39e4-46e0-935c-333653d0615b" (UID: "b6579d9f-39e4-46e0-935c-333653d0615b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:38.396847 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:38.396824 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6579d9f-39e4-46e0-935c-333653d0615b-kube-api-access-qbhnt" pod "b6579d9f-39e4-46e0-935c-333653d0615b" (UID: "b6579d9f-39e4-46e0-935c-333653d0615b"). InnerVolumeSpecName "kube-api-access-qbhnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:38.496234 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:38.496181 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6579d9f-39e4-46e0-935c-333653d0615b-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:38.496234 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:38.496227 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6579d9f-39e4-46e0-935c-333653d0615b-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:38.496234 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:38.496242 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbhnt\" (UniqueName: \"kubernetes.io/projected/b6579d9f-39e4-46e0-935c-333653d0615b-kube-api-access-qbhnt\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:39.138102 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:39.138062 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" event={"ID":"b6579d9f-39e4-46e0-935c-333653d0615b","Type":"ContainerDied","Data":"284b192f5e3e9abaa5bc6940f430749934e95c537e8aa4910756a0b0d8c2c95d"} Nov 03 20:39:39.138102 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:39.138086 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t" Nov 03 20:39:39.138343 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:39.138125 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="284b192f5e3e9abaa5bc6940f430749934e95c537e8aa4910756a0b0d8c2c95d" Nov 03 20:39:40.711680 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:40.711637 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" podUID="6168df8b-cb4f-4e2f-87f4-8a1138ee32ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:39:40.712104 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:40.711644 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" podUID="658bb868-b206-4256-806e-10eb7f091599" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:39:50.711167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:50.711124 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" podUID="658bb868-b206-4256-806e-10eb7f091599" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:39:50.711737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:50.711330 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" podUID="6168df8b-cb4f-4e2f-87f4-8a1138ee32ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:39:51.012018 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.011987 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" Nov 03 20:39:51.064990 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.064960 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" Nov 03 20:39:51.126058 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.126026 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6168df8b-cb4f-4e2f-87f4-8a1138ee32ff-kube-api-access-zmbcr\" (UniqueName: \"kubernetes.io/projected/6168df8b-cb4f-4e2f-87f4-8a1138ee32ff-kube-api-access-zmbcr\") pod \"6168df8b-cb4f-4e2f-87f4-8a1138ee32ff\" (UID: \"6168df8b-cb4f-4e2f-87f4-8a1138ee32ff\") " Nov 03 20:39:51.126221 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.126133 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/658bb868-b206-4256-806e-10eb7f091599-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/658bb868-b206-4256-806e-10eb7f091599-plugins-config-volume\") pod \"658bb868-b206-4256-806e-10eb7f091599\" (UID: \"658bb868-b206-4256-806e-10eb7f091599\") " Nov 03 20:39:51.126221 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.126165 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6168df8b-cb4f-4e2f-87f4-8a1138ee32ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6168df8b-cb4f-4e2f-87f4-8a1138ee32ff-plugins-config-volume\") pod \"6168df8b-cb4f-4e2f-87f4-8a1138ee32ff\" (UID: \"6168df8b-cb4f-4e2f-87f4-8a1138ee32ff\") " Nov 03 20:39:51.126369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.126250 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/658bb868-b206-4256-806e-10eb7f091599-kube-api-access-7ndst\" (UniqueName: \"kubernetes.io/projected/658bb868-b206-4256-806e-10eb7f091599-kube-api-access-7ndst\") pod \"658bb868-b206-4256-806e-10eb7f091599\" (UID: \"658bb868-b206-4256-806e-10eb7f091599\") " Nov 03 20:39:51.126586 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.126517 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/658bb868-b206-4256-806e-10eb7f091599-plugins-config-volume" pod "658bb868-b206-4256-806e-10eb7f091599" (UID: "658bb868-b206-4256-806e-10eb7f091599"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:51.126723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.126585 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6168df8b-cb4f-4e2f-87f4-8a1138ee32ff-plugins-config-volume" pod "6168df8b-cb4f-4e2f-87f4-8a1138ee32ff" (UID: "6168df8b-cb4f-4e2f-87f4-8a1138ee32ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:51.128271 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.128252 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6168df8b-cb4f-4e2f-87f4-8a1138ee32ff-kube-api-access-zmbcr" pod "6168df8b-cb4f-4e2f-87f4-8a1138ee32ff" (UID: "6168df8b-cb4f-4e2f-87f4-8a1138ee32ff"). InnerVolumeSpecName "kube-api-access-zmbcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:51.128326 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.128272 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658bb868-b206-4256-806e-10eb7f091599-kube-api-access-7ndst" pod "658bb868-b206-4256-806e-10eb7f091599" (UID: "658bb868-b206-4256-806e-10eb7f091599"). InnerVolumeSpecName "kube-api-access-7ndst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:51.202866 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.202824 5111 generic.go:356] "Generic (PLEG): container finished" podID="658bb868-b206-4256-806e-10eb7f091599" containerID="59506ee563f53d76a63fc1d19295cddc76a5faa045a3ba71d8b58a15337baf3d" exitCode=0 Nov 03 20:39:51.203080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.202868 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" Nov 03 20:39:51.203080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.202910 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" event={"ID":"658bb868-b206-4256-806e-10eb7f091599","Type":"ContainerDied","Data":"59506ee563f53d76a63fc1d19295cddc76a5faa045a3ba71d8b58a15337baf3d"} Nov 03 20:39:51.203080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.202970 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d" event={"ID":"658bb868-b206-4256-806e-10eb7f091599","Type":"ContainerDied","Data":"85f0b10680517b8c92b5be194371b2c10020cb93cccb71a7abb902a5381d88c5"} Nov 03 20:39:51.203080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.202998 5111 scope.go:122] "RemoveContainer" containerID="59506ee563f53d76a63fc1d19295cddc76a5faa045a3ba71d8b58a15337baf3d" Nov 03 20:39:51.204280 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.204257 5111 generic.go:356] "Generic (PLEG): container finished" podID="6168df8b-cb4f-4e2f-87f4-8a1138ee32ff" containerID="b3fb31ea3692c009ad198124779ecd4abdf10e77a35bc995ae8e0682ba7fff6b" exitCode=0 Nov 03 20:39:51.204400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.204286 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" event={"ID":"6168df8b-cb4f-4e2f-87f4-8a1138ee32ff","Type":"ContainerDied","Data":"b3fb31ea3692c009ad198124779ecd4abdf10e77a35bc995ae8e0682ba7fff6b"} Nov 03 20:39:51.204400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.204304 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" Nov 03 20:39:51.204400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.204321 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn" event={"ID":"6168df8b-cb4f-4e2f-87f4-8a1138ee32ff","Type":"ContainerDied","Data":"a94e2fdd8a82f06c7aa2c64633d30e686af50936a7c316c81cbcc8793498c36f"} Nov 03 20:39:51.212254 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.212236 5111 scope.go:122] "RemoveContainer" containerID="59506ee563f53d76a63fc1d19295cddc76a5faa045a3ba71d8b58a15337baf3d" Nov 03 20:39:51.212506 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:39:51.212489 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59506ee563f53d76a63fc1d19295cddc76a5faa045a3ba71d8b58a15337baf3d\": container with ID starting with 59506ee563f53d76a63fc1d19295cddc76a5faa045a3ba71d8b58a15337baf3d not found: ID does not exist" containerID="59506ee563f53d76a63fc1d19295cddc76a5faa045a3ba71d8b58a15337baf3d" Nov 03 20:39:51.212599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.212514 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59506ee563f53d76a63fc1d19295cddc76a5faa045a3ba71d8b58a15337baf3d"} err="failed to get container status \"59506ee563f53d76a63fc1d19295cddc76a5faa045a3ba71d8b58a15337baf3d\": rpc error: code = NotFound desc = could not find container \"59506ee563f53d76a63fc1d19295cddc76a5faa045a3ba71d8b58a15337baf3d\": container with ID starting with 59506ee563f53d76a63fc1d19295cddc76a5faa045a3ba71d8b58a15337baf3d not found: ID does not exist" Nov 03 20:39:51.212599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.212543 5111 scope.go:122] "RemoveContainer" containerID="b3fb31ea3692c009ad198124779ecd4abdf10e77a35bc995ae8e0682ba7fff6b" Nov 03 20:39:51.221425 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.221408 5111 scope.go:122] "RemoveContainer" containerID="b3fb31ea3692c009ad198124779ecd4abdf10e77a35bc995ae8e0682ba7fff6b" Nov 03 20:39:51.221728 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:39:51.221707 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3fb31ea3692c009ad198124779ecd4abdf10e77a35bc995ae8e0682ba7fff6b\": container with ID starting with b3fb31ea3692c009ad198124779ecd4abdf10e77a35bc995ae8e0682ba7fff6b not found: ID does not exist" containerID="b3fb31ea3692c009ad198124779ecd4abdf10e77a35bc995ae8e0682ba7fff6b" Nov 03 20:39:51.221779 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.221737 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3fb31ea3692c009ad198124779ecd4abdf10e77a35bc995ae8e0682ba7fff6b"} err="failed to get container status \"b3fb31ea3692c009ad198124779ecd4abdf10e77a35bc995ae8e0682ba7fff6b\": rpc error: code = NotFound desc = could not find container \"b3fb31ea3692c009ad198124779ecd4abdf10e77a35bc995ae8e0682ba7fff6b\": container with ID starting with b3fb31ea3692c009ad198124779ecd4abdf10e77a35bc995ae8e0682ba7fff6b not found: ID does not exist" Nov 03 20:39:51.227381 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.227354 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6168df8b-cb4f-4e2f-87f4-8a1138ee32ff-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:51.227497 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.227383 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ndst\" (UniqueName: \"kubernetes.io/projected/658bb868-b206-4256-806e-10eb7f091599-kube-api-access-7ndst\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:51.227497 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.227398 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmbcr\" (UniqueName: \"kubernetes.io/projected/6168df8b-cb4f-4e2f-87f4-8a1138ee32ff-kube-api-access-zmbcr\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:51.227497 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.227414 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/658bb868-b206-4256-806e-10eb7f091599-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:51.230110 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.230086 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d"] Nov 03 20:39:51.234134 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.234074 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4442a4-7cff98b6bzhn8d"] Nov 03 20:39:51.244683 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.244660 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn"] Nov 03 20:39:51.247922 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:51.247898 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4442a4-7658db52vn"] Nov 03 20:39:52.181244 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:52.181199 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6168df8b-cb4f-4e2f-87f4-8a1138ee32ff" path="/var/lib/kubelet/pods/6168df8b-cb4f-4e2f-87f4-8a1138ee32ff/volumes" Nov 03 20:39:52.181824 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:39:52.181800 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="658bb868-b206-4256-806e-10eb7f091599" path="/var/lib/kubelet/pods/658bb868-b206-4256-806e-10eb7f091599/volumes" Nov 03 20:40:03.602614 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.602574 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd"] Nov 03 20:40:03.604158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.604127 5111 cpu_manager.go:414] "containerMap: removing container" podUID="658bb868-b206-4256-806e-10eb7f091599" containerName="payload-processing" Nov 03 20:40:03.604158 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.604158 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="658bb868-b206-4256-806e-10eb7f091599" containerName="payload-processing" Nov 03 20:40:03.604332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.604182 5111 cpu_manager.go:414] "containerMap: removing container" podUID="b6579d9f-39e4-46e0-935c-333653d0615b" containerName="revoke-keys" Nov 03 20:40:03.604332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.604191 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6579d9f-39e4-46e0-935c-333653d0615b" containerName="revoke-keys" Nov 03 20:40:03.604332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.604229 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6168df8b-cb4f-4e2f-87f4-8a1138ee32ff" containerName="payload-pre-processing" Nov 03 20:40:03.604332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.604235 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6168df8b-cb4f-4e2f-87f4-8a1138ee32ff" containerName="payload-pre-processing" Nov 03 20:40:03.604332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.604318 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6579d9f-39e4-46e0-935c-333653d0615b" containerName="revoke-keys" Nov 03 20:40:03.604332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.604332 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6168df8b-cb4f-4e2f-87f4-8a1138ee32ff" containerName="payload-pre-processing" Nov 03 20:40:03.604647 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.604340 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="658bb868-b206-4256-806e-10eb7f091599" containerName="payload-processing" Nov 03 20:40:03.609445 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.609421 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.622627 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.622589 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd"] Nov 03 20:40:03.757409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.757371 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/130f1293-a57b-49b5-8e6a-471751b565f9-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.757609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.757423 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/130f1293-a57b-49b5-8e6a-471751b565f9-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.757609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.757494 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/130f1293-a57b-49b5-8e6a-471751b565f9-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.757609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.757561 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/130f1293-a57b-49b5-8e6a-471751b565f9-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.757801 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.757659 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87n96\" (UniqueName: \"kubernetes.io/projected/130f1293-a57b-49b5-8e6a-471751b565f9-kube-api-access-87n96\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.757801 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.757704 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/130f1293-a57b-49b5-8e6a-471751b565f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.757801 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.757720 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/130f1293-a57b-49b5-8e6a-471751b565f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.757801 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.757741 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/130f1293-a57b-49b5-8e6a-471751b565f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.757936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.757836 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/130f1293-a57b-49b5-8e6a-471751b565f9-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.859236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.859146 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/130f1293-a57b-49b5-8e6a-471751b565f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.859236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.859191 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/130f1293-a57b-49b5-8e6a-471751b565f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.859236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.859222 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/130f1293-a57b-49b5-8e6a-471751b565f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.859558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.859263 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/130f1293-a57b-49b5-8e6a-471751b565f9-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.859558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.859321 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/130f1293-a57b-49b5-8e6a-471751b565f9-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.859558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.859367 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/130f1293-a57b-49b5-8e6a-471751b565f9-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.859558 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.859499 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/130f1293-a57b-49b5-8e6a-471751b565f9-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.859792 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.859609 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/130f1293-a57b-49b5-8e6a-471751b565f9-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.859792 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.859669 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87n96\" (UniqueName: \"kubernetes.io/projected/130f1293-a57b-49b5-8e6a-471751b565f9-kube-api-access-87n96\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.859792 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.859766 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/130f1293-a57b-49b5-8e6a-471751b565f9-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.859959 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.859852 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/130f1293-a57b-49b5-8e6a-471751b565f9-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.860076 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.860055 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/130f1293-a57b-49b5-8e6a-471751b565f9-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.860145 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.860095 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/130f1293-a57b-49b5-8e6a-471751b565f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.860145 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.860114 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/130f1293-a57b-49b5-8e6a-471751b565f9-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.861815 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.861784 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/130f1293-a57b-49b5-8e6a-471751b565f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.862001 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.861981 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/130f1293-a57b-49b5-8e6a-471751b565f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.867258 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.867234 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/130f1293-a57b-49b5-8e6a-471751b565f9-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.867690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.867670 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87n96\" (UniqueName: \"kubernetes.io/projected/130f1293-a57b-49b5-8e6a-471751b565f9-kube-api-access-87n96\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2z5gd\" (UID: \"130f1293-a57b-49b5-8e6a-471751b565f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:03.928940 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:03.928896 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:04.213391 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:04.213228 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd"] Nov 03 20:40:04.215929 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:40:04.215899 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130f1293_a57b_49b5_8e6a_471751b565f9.slice/crio-960e7a1d8f2d25c32e7fffd4abaab263cd236c9a074e1eeb393ccc8789fde602 WatchSource:0}: Error finding container 960e7a1d8f2d25c32e7fffd4abaab263cd236c9a074e1eeb393ccc8789fde602: Status 404 returned error can't find the container with id 960e7a1d8f2d25c32e7fffd4abaab263cd236c9a074e1eeb393ccc8789fde602 Nov 03 20:40:04.218243 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:04.218205 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:40:04.218360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:04.218277 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:40:04.218360 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:04.218306 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:40:04.273107 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:04.273067 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" event={"ID":"130f1293-a57b-49b5-8e6a-471751b565f9","Type":"ContainerStarted","Data":"960e7a1d8f2d25c32e7fffd4abaab263cd236c9a074e1eeb393ccc8789fde602"} Nov 03 20:40:05.058465 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:40:05.058421 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:40:05.279705 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:05.279669 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" event={"ID":"130f1293-a57b-49b5-8e6a-471751b565f9","Type":"ContainerStarted","Data":"a226fee65486cddf8a0484fb2ccd77db5e9a90f8199af5836b1d935a273bb7e6"} Nov 03 20:40:05.930044 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:05.930001 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:05.935458 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:05.935424 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:05.959612 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:05.959550 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" podStartSLOduration=2.9595214199999997 podStartE2EDuration="2.95952142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:05.300936402 +0000 UTC m=+3635.263314842" watchObservedRunningTime="2026-11-03 20:40:05.95952142 +0000 UTC m=+3635.921899859" Nov 03 20:40:06.285172 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:06.285137 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:06.286323 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:06.286299 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:40:18.621933 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.621852 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv"] Nov 03 20:40:18.625765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.625746 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.635837 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.635808 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv"] Nov 03 20:40:18.707282 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.707240 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.707481 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.707295 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.707481 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.707402 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.707481 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.707438 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.707481 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.707466 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdkhs\" (UniqueName: \"kubernetes.io/projected/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-kube-api-access-hdkhs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.707740 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.707627 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.707740 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.707666 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.707740 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.707688 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.707872 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.707776 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.808819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.808770 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.809029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.808841 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.809029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.808871 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.809029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.808903 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.809029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.808965 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.809029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.809015 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.809321 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.809076 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.809321 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.809107 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.809321 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.809150 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdkhs\" (UniqueName: \"kubernetes.io/projected/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-kube-api-access-hdkhs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.809507 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.809393 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.809507 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.809484 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.809787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.809766 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.809857 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.809783 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.809857 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.809832 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.811266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.811237 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.811481 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.811463 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.816803 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.816775 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.816952 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.816934 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdkhs\" (UniqueName: \"kubernetes.io/projected/37ddf8c6-464e-4a46-85f9-4cd0161e5b6c-kube-api-access-hdkhs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-lc2bv\" (UID: \"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:18.944888 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:18.944801 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:19.226419 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:19.226390 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv"] Nov 03 20:40:19.228329 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:40:19.228294 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ddf8c6_464e_4a46_85f9_4cd0161e5b6c.slice/crio-58b74ff1d0db6dffa1c1c983ed0b9154cb7aa80f9d4251d9405839933c002d4b WatchSource:0}: Error finding container 58b74ff1d0db6dffa1c1c983ed0b9154cb7aa80f9d4251d9405839933c002d4b: Status 404 returned error can't find the container with id 58b74ff1d0db6dffa1c1c983ed0b9154cb7aa80f9d4251d9405839933c002d4b Nov 03 20:40:19.230597 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:19.230554 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:40:19.230742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:19.230642 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:40:19.230742 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:19.230690 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:40:19.362428 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:19.362387 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" event={"ID":"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c","Type":"ContainerStarted","Data":"cd4f689e74d4d03c928b726cf105ae50da22c7c6c5fb0eda5bb3a1c7eedf456f"} Nov 03 20:40:19.362640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:19.362450 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" event={"ID":"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c","Type":"ContainerStarted","Data":"58b74ff1d0db6dffa1c1c983ed0b9154cb7aa80f9d4251d9405839933c002d4b"} Nov 03 20:40:19.382608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:19.382326 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" podStartSLOduration=1.38230842 podStartE2EDuration="1.38230842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:19.380720777 +0000 UTC m=+3649.343099218" watchObservedRunningTime="2026-11-03 20:40:19.38230842 +0000 UTC m=+3649.344686860" Nov 03 20:40:19.945160 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:19.945120 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:20.950661 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:20.950618 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:21.372214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:21.372181 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:21.373145 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:40:21.373126 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:40:35.055888 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:40:35.055848 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:41:05.056610 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:41:05.056562 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:41:35.101172 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:41:35.101132 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:41:40.846947 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:40.846858 5111 generic.go:356] "Generic (PLEG): container finished" podID="37ddf8c6-464e-4a46-85f9-4cd0161e5b6c" containerID="cd4f689e74d4d03c928b726cf105ae50da22c7c6c5fb0eda5bb3a1c7eedf456f" exitCode=0 Nov 03 20:41:40.846947 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:40.846934 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" event={"ID":"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c","Type":"ContainerDied","Data":"cd4f689e74d4d03c928b726cf105ae50da22c7c6c5fb0eda5bb3a1c7eedf456f"} Nov 03 20:41:40.847496 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:40.847413 5111 scope.go:122] "RemoveContainer" containerID="cd4f689e74d4d03c928b726cf105ae50da22c7c6c5fb0eda5bb3a1c7eedf456f" Nov 03 20:41:40.848716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:40.848694 5111 generic.go:356] "Generic (PLEG): container finished" podID="130f1293-a57b-49b5-8e6a-471751b565f9" containerID="a226fee65486cddf8a0484fb2ccd77db5e9a90f8199af5836b1d935a273bb7e6" exitCode=0 Nov 03 20:41:40.848821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:40.848734 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:41:40.848821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:40.848760 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" event={"ID":"130f1293-a57b-49b5-8e6a-471751b565f9","Type":"ContainerDied","Data":"a226fee65486cddf8a0484fb2ccd77db5e9a90f8199af5836b1d935a273bb7e6"} Nov 03 20:41:40.848821 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:40.848802 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:41:40.848935 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:40.848835 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:41:40.849101 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:40.849085 5111 scope.go:122] "RemoveContainer" containerID="a226fee65486cddf8a0484fb2ccd77db5e9a90f8199af5836b1d935a273bb7e6" Nov 03 20:41:40.850189 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:40.850161 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:41:40.850272 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:40.850249 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:41:40.850353 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:40.850308 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:41:41.895163 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:41.895123 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" event={"ID":"130f1293-a57b-49b5-8e6a-471751b565f9","Type":"ContainerStarted","Data":"398a6c7cce8ddebb6cd3eb4147d1757d9f1a86edb9181a1a34e99c05741536b6"} Nov 03 20:41:41.898330 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:41.898291 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" event={"ID":"37ddf8c6-464e-4a46-85f9-4cd0161e5b6c","Type":"ContainerStarted","Data":"488f95acfcd1610430da3c4c963899bc710a55dd2445d9d99a5bf70d43080dbf"} Nov 03 20:41:41.929907 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:41.929875 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:41:41.930200 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:41.930167 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2z5gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Nov 03 20:41:41.930339 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:41.930214 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" podUID="130f1293-a57b-49b5-8e6a-471751b565f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Nov 03 20:41:41.970269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:41.970227 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:41:41.980730 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:41.980685 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lc2bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 20:41:41.981599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:41.981561 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" podUID="37ddf8c6-464e-4a46-85f9-4cd0161e5b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 20:41:42.929355 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:42.929170 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2z5gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Nov 03 20:41:42.929355 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:42.929235 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" podUID="130f1293-a57b-49b5-8e6a-471751b565f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Nov 03 20:41:42.952557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:42.948856 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lc2bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 20:41:42.952557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:42.948909 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" podUID="37ddf8c6-464e-4a46-85f9-4cd0161e5b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 20:41:43.898734 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:43.898700 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64"] Nov 03 20:41:43.901617 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:43.901595 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64" Nov 03 20:41:43.905706 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:43.905679 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:41:43.905850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:43.905723 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:41:43.906636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:43.906573 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:41:43.906636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:43.906617 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:41:43.909864 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:43.909837 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64"] Nov 03 20:41:43.930335 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:43.930293 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2z5gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Nov 03 20:41:43.930790 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:43.930369 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" podUID="130f1293-a57b-49b5-8e6a-471751b565f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Nov 03 20:41:43.970026 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:43.969981 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lc2bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 20:41:43.974045 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:43.973597 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" podUID="37ddf8c6-464e-4a46-85f9-4cd0161e5b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 20:41:44.022876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:44.022836 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk45s\" (UniqueName: \"kubernetes.io/projected/efb95ec1-e019-477c-bbec-0ce699cd3fac-kube-api-access-qk45s\") pod \"maas-api-revoke-sub-d576e68b-jbf64\" (UID: \"efb95ec1-e019-477c-bbec-0ce699cd3fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64" Nov 03 20:41:44.023066 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:44.022887 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb95ec1-e019-477c-bbec-0ce699cd3fac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-jbf64\" (UID: \"efb95ec1-e019-477c-bbec-0ce699cd3fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64" Nov 03 20:41:44.023066 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:44.023015 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb95ec1-e019-477c-bbec-0ce699cd3fac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-jbf64\" (UID: \"efb95ec1-e019-477c-bbec-0ce699cd3fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64" Nov 03 20:41:44.123907 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:44.123867 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb95ec1-e019-477c-bbec-0ce699cd3fac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-jbf64\" (UID: \"efb95ec1-e019-477c-bbec-0ce699cd3fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64" Nov 03 20:41:44.124131 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:44.124004 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk45s\" (UniqueName: \"kubernetes.io/projected/efb95ec1-e019-477c-bbec-0ce699cd3fac-kube-api-access-qk45s\") pod \"maas-api-revoke-sub-d576e68b-jbf64\" (UID: \"efb95ec1-e019-477c-bbec-0ce699cd3fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64" Nov 03 20:41:44.124131 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:44.124048 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb95ec1-e019-477c-bbec-0ce699cd3fac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-jbf64\" (UID: \"efb95ec1-e019-477c-bbec-0ce699cd3fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64" Nov 03 20:41:44.124834 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:44.124808 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb95ec1-e019-477c-bbec-0ce699cd3fac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-jbf64\" (UID: \"efb95ec1-e019-477c-bbec-0ce699cd3fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64" Nov 03 20:41:44.126556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:44.126521 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb95ec1-e019-477c-bbec-0ce699cd3fac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-jbf64\" (UID: \"efb95ec1-e019-477c-bbec-0ce699cd3fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64" Nov 03 20:41:44.133280 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:44.133255 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk45s\" (UniqueName: \"kubernetes.io/projected/efb95ec1-e019-477c-bbec-0ce699cd3fac-kube-api-access-qk45s\") pod \"maas-api-revoke-sub-d576e68b-jbf64\" (UID: \"efb95ec1-e019-477c-bbec-0ce699cd3fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64" Nov 03 20:41:44.215877 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:44.215795 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64" Nov 03 20:41:44.536599 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:44.536570 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64"] Nov 03 20:41:44.538667 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:41:44.538634 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb95ec1_e019_477c_bbec_0ce699cd3fac.slice/crio-7a1dde46e8aa221912eeb4a86b0d906ad71d86156a7687cf28caeaef9590d5c7 WatchSource:0}: Error finding container 7a1dde46e8aa221912eeb4a86b0d906ad71d86156a7687cf28caeaef9590d5c7: Status 404 returned error can't find the container with id 7a1dde46e8aa221912eeb4a86b0d906ad71d86156a7687cf28caeaef9590d5c7 Nov 03 20:41:44.925445 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:44.925404 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64" event={"ID":"efb95ec1-e019-477c-bbec-0ce699cd3fac","Type":"ContainerStarted","Data":"7a1dde46e8aa221912eeb4a86b0d906ad71d86156a7687cf28caeaef9590d5c7"} Nov 03 20:41:44.929764 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:44.929737 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2z5gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Nov 03 20:41:44.929924 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:44.929792 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" podUID="130f1293-a57b-49b5-8e6a-471751b565f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Nov 03 20:41:44.956115 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:44.956080 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lc2bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 20:41:44.961666 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:44.961631 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" podUID="37ddf8c6-464e-4a46-85f9-4cd0161e5b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 20:41:45.929293 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:45.929240 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2z5gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Nov 03 20:41:45.929482 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:45.929306 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" podUID="130f1293-a57b-49b5-8e6a-471751b565f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Nov 03 20:41:45.931795 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:45.931769 5111 generic.go:356] "Generic (PLEG): container finished" podID="efb95ec1-e019-477c-bbec-0ce699cd3fac" containerID="66ac2cdc75f7352ab2ab13aa9c294f54a3e06ce05ef686b91dac2771293ea5e8" exitCode=0 Nov 03 20:41:45.931936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:45.931843 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64" event={"ID":"efb95ec1-e019-477c-bbec-0ce699cd3fac","Type":"ContainerDied","Data":"66ac2cdc75f7352ab2ab13aa9c294f54a3e06ce05ef686b91dac2771293ea5e8"} Nov 03 20:41:45.971557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:45.971198 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lc2bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 20:41:45.971557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:45.971290 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" podUID="37ddf8c6-464e-4a46-85f9-4cd0161e5b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 20:41:46.930232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:46.930184 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2z5gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Nov 03 20:41:46.930511 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:46.930260 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" podUID="130f1293-a57b-49b5-8e6a-471751b565f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Nov 03 20:41:46.945780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:46.945743 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lc2bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 20:41:46.945927 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:46.945821 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" podUID="37ddf8c6-464e-4a46-85f9-4cd0161e5b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 20:41:47.100854 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.100823 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64" Nov 03 20:41:47.207052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.206954 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb95ec1-e019-477c-bbec-0ce699cd3fac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb95ec1-e019-477c-bbec-0ce699cd3fac-maas-api-openshift-ca\") pod \"efb95ec1-e019-477c-bbec-0ce699cd3fac\" (UID: \"efb95ec1-e019-477c-bbec-0ce699cd3fac\") " Nov 03 20:41:47.207052 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.207041 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efb95ec1-e019-477c-bbec-0ce699cd3fac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb95ec1-e019-477c-bbec-0ce699cd3fac-maas-api-xks-ca\") pod \"efb95ec1-e019-477c-bbec-0ce699cd3fac\" (UID: \"efb95ec1-e019-477c-bbec-0ce699cd3fac\") " Nov 03 20:41:47.207267 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.207218 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb95ec1-e019-477c-bbec-0ce699cd3fac-kube-api-access-qk45s\" (UniqueName: \"kubernetes.io/projected/efb95ec1-e019-477c-bbec-0ce699cd3fac-kube-api-access-qk45s\") pod \"efb95ec1-e019-477c-bbec-0ce699cd3fac\" (UID: \"efb95ec1-e019-477c-bbec-0ce699cd3fac\") " Nov 03 20:41:47.207420 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.207399 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb95ec1-e019-477c-bbec-0ce699cd3fac-maas-api-openshift-ca" pod "efb95ec1-e019-477c-bbec-0ce699cd3fac" (UID: "efb95ec1-e019-477c-bbec-0ce699cd3fac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:47.207616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.207595 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb95ec1-e019-477c-bbec-0ce699cd3fac-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:47.209412 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.209376 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb95ec1-e019-477c-bbec-0ce699cd3fac-maas-api-xks-ca" pod "efb95ec1-e019-477c-bbec-0ce699cd3fac" (UID: "efb95ec1-e019-477c-bbec-0ce699cd3fac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:41:47.209580 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.209498 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb95ec1-e019-477c-bbec-0ce699cd3fac-kube-api-access-qk45s" pod "efb95ec1-e019-477c-bbec-0ce699cd3fac" (UID: "efb95ec1-e019-477c-bbec-0ce699cd3fac"). InnerVolumeSpecName "kube-api-access-qk45s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:47.309333 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.309279 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk45s\" (UniqueName: \"kubernetes.io/projected/efb95ec1-e019-477c-bbec-0ce699cd3fac-kube-api-access-qk45s\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:47.309333 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.309326 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb95ec1-e019-477c-bbec-0ce699cd3fac-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:47.929728 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.929682 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2z5gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Nov 03 20:41:47.929939 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.929757 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" podUID="130f1293-a57b-49b5-8e6a-471751b565f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Nov 03 20:41:47.945126 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.945093 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lc2bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 20:41:47.945296 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.945150 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" podUID="37ddf8c6-464e-4a46-85f9-4cd0161e5b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 20:41:47.946400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.946366 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64" event={"ID":"efb95ec1-e019-477c-bbec-0ce699cd3fac","Type":"ContainerDied","Data":"7a1dde46e8aa221912eeb4a86b0d906ad71d86156a7687cf28caeaef9590d5c7"} Nov 03 20:41:47.946527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.946424 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a1dde46e8aa221912eeb4a86b0d906ad71d86156a7687cf28caeaef9590d5c7" Nov 03 20:41:47.946527 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:47.946369 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64" Nov 03 20:41:49.212491 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:49.212433 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:41:49.213397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:49.213375 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:41:49.213975 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:49.213952 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:41:49.214193 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:49.214176 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" Nov 03 20:41:49.214291 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:49.214218 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:41:49.215104 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:41:49.215074 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" Nov 03 20:42:02.731331 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.731295 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66"] Nov 03 20:42:02.731809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.731769 5111 cpu_manager.go:414] "containerMap: removing container" podUID="efb95ec1-e019-477c-bbec-0ce699cd3fac" containerName="revoke-keys" Nov 03 20:42:02.731809 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.731784 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb95ec1-e019-477c-bbec-0ce699cd3fac" containerName="revoke-keys" Nov 03 20:42:02.731907 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.731896 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb95ec1-e019-477c-bbec-0ce699cd3fac" containerName="revoke-keys" Nov 03 20:42:02.735349 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.735330 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" Nov 03 20:42:02.738942 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.738918 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:42:02.739916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.739895 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:42:02.740017 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.739909 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:42:02.740017 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.740004 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:42:02.745486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.745464 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66"] Nov 03 20:42:02.818025 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.817988 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-spn66\" (UID: \"e29b5fc7-ca2a-4985-a1fe-7350a565f86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" Nov 03 20:42:02.818216 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.818119 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-spn66\" (UID: \"e29b5fc7-ca2a-4985-a1fe-7350a565f86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" Nov 03 20:42:02.818216 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.818195 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drdj9\" (UniqueName: \"kubernetes.io/projected/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-kube-api-access-drdj9\") pod \"maas-api-revoke-sub-d2bc93bc-spn66\" (UID: \"e29b5fc7-ca2a-4985-a1fe-7350a565f86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" Nov 03 20:42:02.919836 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.919791 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-spn66\" (UID: \"e29b5fc7-ca2a-4985-a1fe-7350a565f86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" Nov 03 20:42:02.920043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.919933 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drdj9\" (UniqueName: \"kubernetes.io/projected/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-kube-api-access-drdj9\") pod \"maas-api-revoke-sub-d2bc93bc-spn66\" (UID: \"e29b5fc7-ca2a-4985-a1fe-7350a565f86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" Nov 03 20:42:02.920118 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.920050 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-spn66\" (UID: \"e29b5fc7-ca2a-4985-a1fe-7350a565f86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" Nov 03 20:42:02.920713 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.920690 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-spn66\" (UID: \"e29b5fc7-ca2a-4985-a1fe-7350a565f86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" Nov 03 20:42:02.922217 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.922184 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-spn66\" (UID: \"e29b5fc7-ca2a-4985-a1fe-7350a565f86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" Nov 03 20:42:02.935071 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:02.935044 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drdj9\" (UniqueName: \"kubernetes.io/projected/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-kube-api-access-drdj9\") pod \"maas-api-revoke-sub-d2bc93bc-spn66\" (UID: \"e29b5fc7-ca2a-4985-a1fe-7350a565f86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" Nov 03 20:42:03.050526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:03.050431 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" Nov 03 20:42:03.332526 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:03.332495 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66"] Nov 03 20:42:03.335167 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:42:03.335138 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29b5fc7_ca2a_4985_a1fe_7350a565f86f.slice/crio-b2f61aa29355099cff15cc2650c311e324bde7ff705fd2843c9526db8ec02019 WatchSource:0}: Error finding container b2f61aa29355099cff15cc2650c311e324bde7ff705fd2843c9526db8ec02019: Status 404 returned error can't find the container with id b2f61aa29355099cff15cc2650c311e324bde7ff705fd2843c9526db8ec02019 Nov 03 20:42:04.058137 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:04.058046 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" event={"ID":"e29b5fc7-ca2a-4985-a1fe-7350a565f86f","Type":"ContainerStarted","Data":"834cb6dfd5d76c3434aaa5aa2c13a835e9aa46d5e152c2041c0a9ff20273c683"} Nov 03 20:42:04.058137 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:04.058102 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" event={"ID":"e29b5fc7-ca2a-4985-a1fe-7350a565f86f","Type":"ContainerStarted","Data":"b2f61aa29355099cff15cc2650c311e324bde7ff705fd2843c9526db8ec02019"} Nov 03 20:42:04.075061 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:04.075011 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" podStartSLOduration=2.074998095 podStartE2EDuration="2.074998095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:04.073790151 +0000 UTC m=+3754.036168591" watchObservedRunningTime="2026-11-03 20:42:04.074998095 +0000 UTC m=+3754.037376528" Nov 03 20:42:05.055681 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:42:05.055634 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:42:05.064522 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:05.064496 5111 generic.go:356] "Generic (PLEG): container finished" podID="e29b5fc7-ca2a-4985-a1fe-7350a565f86f" containerID="834cb6dfd5d76c3434aaa5aa2c13a835e9aa46d5e152c2041c0a9ff20273c683" exitCode=0 Nov 03 20:42:05.064888 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:05.064574 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" event={"ID":"e29b5fc7-ca2a-4985-a1fe-7350a565f86f","Type":"ContainerDied","Data":"834cb6dfd5d76c3434aaa5aa2c13a835e9aa46d5e152c2041c0a9ff20273c683"} Nov 03 20:42:06.215948 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:06.215923 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" Nov 03 20:42:06.358585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:06.358478 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-maas-api-openshift-ca\") pod \"e29b5fc7-ca2a-4985-a1fe-7350a565f86f\" (UID: \"e29b5fc7-ca2a-4985-a1fe-7350a565f86f\") " Nov 03 20:42:06.358748 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:06.358604 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-maas-api-xks-ca\") pod \"e29b5fc7-ca2a-4985-a1fe-7350a565f86f\" (UID: \"e29b5fc7-ca2a-4985-a1fe-7350a565f86f\") " Nov 03 20:42:06.358748 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:06.358654 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-kube-api-access-drdj9\" (UniqueName: \"kubernetes.io/projected/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-kube-api-access-drdj9\") pod \"e29b5fc7-ca2a-4985-a1fe-7350a565f86f\" (UID: \"e29b5fc7-ca2a-4985-a1fe-7350a565f86f\") " Nov 03 20:42:06.358869 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:06.358839 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-maas-api-openshift-ca" pod "e29b5fc7-ca2a-4985-a1fe-7350a565f86f" (UID: "e29b5fc7-ca2a-4985-a1fe-7350a565f86f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:42:06.360782 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:06.360761 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-maas-api-xks-ca" pod "e29b5fc7-ca2a-4985-a1fe-7350a565f86f" (UID: "e29b5fc7-ca2a-4985-a1fe-7350a565f86f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:42:06.360916 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:06.360897 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-kube-api-access-drdj9" pod "e29b5fc7-ca2a-4985-a1fe-7350a565f86f" (UID: "e29b5fc7-ca2a-4985-a1fe-7350a565f86f"). InnerVolumeSpecName "kube-api-access-drdj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:42:06.459918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:06.459882 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:06.459918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:06.459915 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:06.459918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:06.459925 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drdj9\" (UniqueName: \"kubernetes.io/projected/e29b5fc7-ca2a-4985-a1fe-7350a565f86f-kube-api-access-drdj9\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:07.076458 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:07.076422 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" event={"ID":"e29b5fc7-ca2a-4985-a1fe-7350a565f86f","Type":"ContainerDied","Data":"b2f61aa29355099cff15cc2650c311e324bde7ff705fd2843c9526db8ec02019"} Nov 03 20:42:07.076702 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:07.076478 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2f61aa29355099cff15cc2650c311e324bde7ff705fd2843c9526db8ec02019" Nov 03 20:42:07.076702 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:07.076434 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66" Nov 03 20:42:19.055377 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.055337 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94"] Nov 03 20:42:19.055896 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.055811 5111 cpu_manager.go:414] "containerMap: removing container" podUID="e29b5fc7-ca2a-4985-a1fe-7350a565f86f" containerName="revoke-keys" Nov 03 20:42:19.055896 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.055824 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29b5fc7-ca2a-4985-a1fe-7350a565f86f" containerName="revoke-keys" Nov 03 20:42:19.055983 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.055925 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29b5fc7-ca2a-4985-a1fe-7350a565f86f" containerName="revoke-keys" Nov 03 20:42:19.059640 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.059619 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:42:19.072415 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.071622 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:42:19.072415 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.071984 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:42:19.072723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.072453 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-4b0085-serving-cert\"" Nov 03 20:42:19.073265 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.073242 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7bzpm\"" Nov 03 20:42:19.073401 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.073298 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:42:19.074929 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.074908 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94"] Nov 03 20:42:19.101470 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.101409 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94\" (UID: \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:42:19.101470 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.101466 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr82c\" (UniqueName: \"kubernetes.io/projected/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-kube-api-access-qr82c\") pod \"maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94\" (UID: \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:42:19.101825 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.101578 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94\" (UID: \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:42:19.101825 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.101617 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94\" (UID: \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:42:19.191616 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.191581 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8"] Nov 03 20:42:19.200317 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.200290 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" Nov 03 20:42:19.202114 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.202085 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94\" (UID: \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:42:19.202288 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.202140 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94\" (UID: \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:42:19.202288 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.202210 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea4dc23c-54ee-4768-9e11-c1ec9e413f84-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8\" (UID: \"ea4dc23c-54ee-4768-9e11-c1ec9e413f84\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" Nov 03 20:42:19.202288 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.202264 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl8mk\" (UniqueName: \"kubernetes.io/projected/ea4dc23c-54ee-4768-9e11-c1ec9e413f84-kube-api-access-wl8mk\") pod \"payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8\" (UID: \"ea4dc23c-54ee-4768-9e11-c1ec9e413f84\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" Nov 03 20:42:19.202463 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.202427 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94\" (UID: \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:42:19.202550 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.202466 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr82c\" (UniqueName: \"kubernetes.io/projected/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-kube-api-access-qr82c\") pod \"maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94\" (UID: \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:42:19.202814 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.202792 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94\" (UID: \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:42:19.205659 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.205585 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94\" (UID: \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:42:19.205773 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.205697 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94\" (UID: \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:42:19.207448 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.207418 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8"] Nov 03 20:42:19.208030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.208012 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4b0085\"" Nov 03 20:42:19.215960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.215929 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr82c\" (UniqueName: \"kubernetes.io/projected/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-kube-api-access-qr82c\") pod \"maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94\" (UID: \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:42:19.304596 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.304508 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea4dc23c-54ee-4768-9e11-c1ec9e413f84-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8\" (UID: \"ea4dc23c-54ee-4768-9e11-c1ec9e413f84\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" Nov 03 20:42:19.304805 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.304641 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl8mk\" (UniqueName: \"kubernetes.io/projected/ea4dc23c-54ee-4768-9e11-c1ec9e413f84-kube-api-access-wl8mk\") pod \"payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8\" (UID: \"ea4dc23c-54ee-4768-9e11-c1ec9e413f84\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" Nov 03 20:42:19.307829 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.307752 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea4dc23c-54ee-4768-9e11-c1ec9e413f84-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8\" (UID: \"ea4dc23c-54ee-4768-9e11-c1ec9e413f84\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" Nov 03 20:42:19.318723 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.318690 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl8mk\" (UniqueName: \"kubernetes.io/projected/ea4dc23c-54ee-4768-9e11-c1ec9e413f84-kube-api-access-wl8mk\") pod \"payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8\" (UID: \"ea4dc23c-54ee-4768-9e11-c1ec9e413f84\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" Nov 03 20:42:19.375908 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.375866 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:42:19.540339 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.540301 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" Nov 03 20:42:19.659247 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.659215 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94"] Nov 03 20:42:19.661160 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:42:19.661130 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cbd028_d1be_46fc_bcaa_21bcbc677af2.slice/crio-a87bf6d936a32675ccf225cecee77a0e614d91fc24f5088b3bb1ccd59c60afbc WatchSource:0}: Error finding container a87bf6d936a32675ccf225cecee77a0e614d91fc24f5088b3bb1ccd59c60afbc: Status 404 returned error can't find the container with id a87bf6d936a32675ccf225cecee77a0e614d91fc24f5088b3bb1ccd59c60afbc Nov 03 20:42:19.818623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:19.818579 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8"] Nov 03 20:42:19.823047 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:42:19.823018 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4dc23c_54ee_4768_9e11_c1ec9e413f84.slice/crio-ab6d81194ce469e0602a902b2894334150c1c991269db4eefe69aaa6e6ead807 WatchSource:0}: Error finding container ab6d81194ce469e0602a902b2894334150c1c991269db4eefe69aaa6e6ead807: Status 404 returned error can't find the container with id ab6d81194ce469e0602a902b2894334150c1c991269db4eefe69aaa6e6ead807 Nov 03 20:42:20.148890 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:20.148860 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" event={"ID":"b4cbd028-d1be-46fc-bcaa-21bcbc677af2","Type":"ContainerStarted","Data":"a87bf6d936a32675ccf225cecee77a0e614d91fc24f5088b3bb1ccd59c60afbc"} Nov 03 20:42:20.150472 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:20.150447 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" event={"ID":"ea4dc23c-54ee-4768-9e11-c1ec9e413f84","Type":"ContainerStarted","Data":"0a68032e4bbdb3e592b7de04ca12d335560ebb01a76d319721f4f32fecafb921"} Nov 03 20:42:20.150656 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:20.150496 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" event={"ID":"ea4dc23c-54ee-4768-9e11-c1ec9e413f84","Type":"ContainerStarted","Data":"ab6d81194ce469e0602a902b2894334150c1c991269db4eefe69aaa6e6ead807"} Nov 03 20:42:20.150732 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:20.150695 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" Nov 03 20:42:20.169473 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:20.169413 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" podStartSLOduration=1.169398076 podStartE2EDuration="1.169398076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:20.167648149 +0000 UTC m=+3770.130026587" watchObservedRunningTime="2026-11-03 20:42:20.169398076 +0000 UTC m=+3770.131776515" Nov 03 20:42:21.158884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:21.158841 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" event={"ID":"b4cbd028-d1be-46fc-bcaa-21bcbc677af2","Type":"ContainerStarted","Data":"846fff092077658981ea6984cc387bc32ea801cf4f10b782b98e169c27e8c4b9"} Nov 03 20:42:21.159384 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:21.159155 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:42:21.180232 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:21.180176 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" podStartSLOduration=1.758876602 podStartE2EDuration="2.180161894s" totalImagesPullingTime="421.285292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:21.17926364 +0000 UTC m=+3771.141642078" watchObservedRunningTime="2026-11-03 20:42:21.180161894 +0000 UTC m=+3771.142540333" Nov 03 20:42:27.170024 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:27.169990 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:42:31.162607 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:42:31.162575 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" Nov 03 20:42:35.057115 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:42:35.057076 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:43:05.057155 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:43:05.057117 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:43:25.808825 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:25.808720 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8"] Nov 03 20:43:26.486804 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:26.486762 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94"] Nov 03 20:43:26.487053 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:26.487002 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" podUID="b4cbd028-d1be-46fc-bcaa-21bcbc677af2" containerName="maas-api" containerID="cri-o://846fff092077658981ea6984cc387bc32ea801cf4f10b782b98e169c27e8c4b9" gracePeriod=30 Nov 03 20:43:26.784924 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:26.784894 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:43:26.950608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:26.950521 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-kube-api-access-qr82c\" (UniqueName: \"kubernetes.io/projected/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-kube-api-access-qr82c\") pod \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\" (UID: \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\") " Nov 03 20:43:26.951129 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:26.950621 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-maas-api-tls\") pod \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\" (UID: \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\") " Nov 03 20:43:26.951129 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:26.950656 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-metrics-certs\") pod \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\" (UID: \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\") " Nov 03 20:43:26.951129 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:26.950679 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-openshift-service-ca\") pod \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\" (UID: \"b4cbd028-d1be-46fc-bcaa-21bcbc677af2\") " Nov 03 20:43:26.951268 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:26.951145 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-openshift-service-ca" pod "b4cbd028-d1be-46fc-bcaa-21bcbc677af2" (UID: "b4cbd028-d1be-46fc-bcaa-21bcbc677af2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:43:26.952950 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:26.952925 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-metrics-certs" pod "b4cbd028-d1be-46fc-bcaa-21bcbc677af2" (UID: "b4cbd028-d1be-46fc-bcaa-21bcbc677af2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:43:26.953039 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:26.952948 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-maas-api-tls" pod "b4cbd028-d1be-46fc-bcaa-21bcbc677af2" (UID: "b4cbd028-d1be-46fc-bcaa-21bcbc677af2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:43:26.953098 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:26.953059 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-kube-api-access-qr82c" pod "b4cbd028-d1be-46fc-bcaa-21bcbc677af2" (UID: "b4cbd028-d1be-46fc-bcaa-21bcbc677af2"). InnerVolumeSpecName "kube-api-access-qr82c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:43:27.051692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:27.051605 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr82c\" (UniqueName: \"kubernetes.io/projected/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-kube-api-access-qr82c\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:27.051692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:27.051640 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-maas-api-tls\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:27.051692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:27.051650 5111 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-metrics-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:27.051692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:27.051661 5111 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4cbd028-d1be-46fc-bcaa-21bcbc677af2-openshift-service-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:27.530953 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:27.530913 5111 generic.go:356] "Generic (PLEG): container finished" podID="b4cbd028-d1be-46fc-bcaa-21bcbc677af2" containerID="846fff092077658981ea6984cc387bc32ea801cf4f10b782b98e169c27e8c4b9" exitCode=0 Nov 03 20:43:27.530953 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:27.530962 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" Nov 03 20:43:27.531245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:27.531001 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" event={"ID":"b4cbd028-d1be-46fc-bcaa-21bcbc677af2","Type":"ContainerDied","Data":"846fff092077658981ea6984cc387bc32ea801cf4f10b782b98e169c27e8c4b9"} Nov 03 20:43:27.531245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:27.531061 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94" event={"ID":"b4cbd028-d1be-46fc-bcaa-21bcbc677af2","Type":"ContainerDied","Data":"a87bf6d936a32675ccf225cecee77a0e614d91fc24f5088b3bb1ccd59c60afbc"} Nov 03 20:43:27.531245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:27.531086 5111 scope.go:122] "RemoveContainer" containerID="846fff092077658981ea6984cc387bc32ea801cf4f10b782b98e169c27e8c4b9" Nov 03 20:43:27.542186 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:27.542166 5111 scope.go:122] "RemoveContainer" containerID="846fff092077658981ea6984cc387bc32ea801cf4f10b782b98e169c27e8c4b9" Nov 03 20:43:27.542491 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:43:27.542474 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846fff092077658981ea6984cc387bc32ea801cf4f10b782b98e169c27e8c4b9\": container with ID starting with 846fff092077658981ea6984cc387bc32ea801cf4f10b782b98e169c27e8c4b9 not found: ID does not exist" containerID="846fff092077658981ea6984cc387bc32ea801cf4f10b782b98e169c27e8c4b9" Nov 03 20:43:27.542566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:27.542502 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846fff092077658981ea6984cc387bc32ea801cf4f10b782b98e169c27e8c4b9"} err="failed to get container status \"846fff092077658981ea6984cc387bc32ea801cf4f10b782b98e169c27e8c4b9\": rpc error: code = NotFound desc = could not find container \"846fff092077658981ea6984cc387bc32ea801cf4f10b782b98e169c27e8c4b9\": container with ID starting with 846fff092077658981ea6984cc387bc32ea801cf4f10b782b98e169c27e8c4b9 not found: ID does not exist" Nov 03 20:43:27.554684 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:27.554646 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94"] Nov 03 20:43:27.558566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:27.558513 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b0085-8697876c6c-wzz94"] Nov 03 20:43:28.182234 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:28.182194 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4cbd028-d1be-46fc-bcaa-21bcbc677af2" path="/var/lib/kubelet/pods/b4cbd028-d1be-46fc-bcaa-21bcbc677af2/volumes" Nov 03 20:43:30.810147 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:30.810094 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" podUID="ea4dc23c-54ee-4768-9e11-c1ec9e413f84" containerName="payload-pre-processing" containerID="cri-o://0a68032e4bbdb3e592b7de04ca12d335560ebb01a76d319721f4f32fecafb921" gracePeriod=25 Nov 03 20:43:31.082294 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:31.082216 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" Nov 03 20:43:31.195858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:31.195818 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4dc23c-54ee-4768-9e11-c1ec9e413f84-kube-api-access-wl8mk\" (UniqueName: \"kubernetes.io/projected/ea4dc23c-54ee-4768-9e11-c1ec9e413f84-kube-api-access-wl8mk\") pod \"ea4dc23c-54ee-4768-9e11-c1ec9e413f84\" (UID: \"ea4dc23c-54ee-4768-9e11-c1ec9e413f84\") " Nov 03 20:43:31.195858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:31.195867 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea4dc23c-54ee-4768-9e11-c1ec9e413f84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea4dc23c-54ee-4768-9e11-c1ec9e413f84-plugins-config-volume\") pod \"ea4dc23c-54ee-4768-9e11-c1ec9e413f84\" (UID: \"ea4dc23c-54ee-4768-9e11-c1ec9e413f84\") " Nov 03 20:43:31.196287 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:31.196262 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea4dc23c-54ee-4768-9e11-c1ec9e413f84-plugins-config-volume" pod "ea4dc23c-54ee-4768-9e11-c1ec9e413f84" (UID: "ea4dc23c-54ee-4768-9e11-c1ec9e413f84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:43:31.198107 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:31.198084 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4dc23c-54ee-4768-9e11-c1ec9e413f84-kube-api-access-wl8mk" pod "ea4dc23c-54ee-4768-9e11-c1ec9e413f84" (UID: "ea4dc23c-54ee-4768-9e11-c1ec9e413f84"). InnerVolumeSpecName "kube-api-access-wl8mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:43:31.297444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:31.297404 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl8mk\" (UniqueName: \"kubernetes.io/projected/ea4dc23c-54ee-4768-9e11-c1ec9e413f84-kube-api-access-wl8mk\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:31.297444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:31.297443 5111 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea4dc23c-54ee-4768-9e11-c1ec9e413f84-plugins-config-volume\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:31.555109 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:31.555076 5111 generic.go:356] "Generic (PLEG): container finished" podID="ea4dc23c-54ee-4768-9e11-c1ec9e413f84" containerID="0a68032e4bbdb3e592b7de04ca12d335560ebb01a76d319721f4f32fecafb921" exitCode=0 Nov 03 20:43:31.555324 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:31.555124 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" Nov 03 20:43:31.555324 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:31.555161 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" event={"ID":"ea4dc23c-54ee-4768-9e11-c1ec9e413f84","Type":"ContainerDied","Data":"0a68032e4bbdb3e592b7de04ca12d335560ebb01a76d319721f4f32fecafb921"} Nov 03 20:43:31.555324 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:31.555218 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8" event={"ID":"ea4dc23c-54ee-4768-9e11-c1ec9e413f84","Type":"ContainerDied","Data":"ab6d81194ce469e0602a902b2894334150c1c991269db4eefe69aaa6e6ead807"} Nov 03 20:43:31.555324 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:31.555241 5111 scope.go:122] "RemoveContainer" containerID="0a68032e4bbdb3e592b7de04ca12d335560ebb01a76d319721f4f32fecafb921" Nov 03 20:43:31.565637 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:31.565613 5111 scope.go:122] "RemoveContainer" containerID="0a68032e4bbdb3e592b7de04ca12d335560ebb01a76d319721f4f32fecafb921" Nov 03 20:43:31.565967 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:43:31.565945 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a68032e4bbdb3e592b7de04ca12d335560ebb01a76d319721f4f32fecafb921\": container with ID starting with 0a68032e4bbdb3e592b7de04ca12d335560ebb01a76d319721f4f32fecafb921 not found: ID does not exist" containerID="0a68032e4bbdb3e592b7de04ca12d335560ebb01a76d319721f4f32fecafb921" Nov 03 20:43:31.566024 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:31.565977 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a68032e4bbdb3e592b7de04ca12d335560ebb01a76d319721f4f32fecafb921"} err="failed to get container status \"0a68032e4bbdb3e592b7de04ca12d335560ebb01a76d319721f4f32fecafb921\": rpc error: code = NotFound desc = could not find container \"0a68032e4bbdb3e592b7de04ca12d335560ebb01a76d319721f4f32fecafb921\": container with ID starting with 0a68032e4bbdb3e592b7de04ca12d335560ebb01a76d319721f4f32fecafb921 not found: ID does not exist" Nov 03 20:43:31.579265 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:31.579219 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8"] Nov 03 20:43:31.584227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:31.584196 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b0085-7f8d5dtqd8"] Nov 03 20:43:32.182692 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:32.182656 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea4dc23c-54ee-4768-9e11-c1ec9e413f84" path="/var/lib/kubelet/pods/ea4dc23c-54ee-4768-9e11-c1ec9e413f84/volumes" Nov 03 20:43:35.057368 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:43:35.057324 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:43:37.991298 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:37.991259 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj"] Nov 03 20:43:37.991998 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:37.991975 5111 cpu_manager.go:414] "containerMap: removing container" podUID="ea4dc23c-54ee-4768-9e11-c1ec9e413f84" containerName="payload-pre-processing" Nov 03 20:43:37.992048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:37.992004 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea4dc23c-54ee-4768-9e11-c1ec9e413f84" containerName="payload-pre-processing" Nov 03 20:43:37.992048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:37.992024 5111 cpu_manager.go:414] "containerMap: removing container" podUID="b4cbd028-d1be-46fc-bcaa-21bcbc677af2" containerName="maas-api" Nov 03 20:43:37.992048 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:37.992032 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4cbd028-d1be-46fc-bcaa-21bcbc677af2" containerName="maas-api" Nov 03 20:43:37.992214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:37.992194 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea4dc23c-54ee-4768-9e11-c1ec9e413f84" containerName="payload-pre-processing" Nov 03 20:43:37.992250 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:37.992213 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4cbd028-d1be-46fc-bcaa-21bcbc677af2" containerName="maas-api" Nov 03 20:43:37.995818 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:37.995795 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.000351 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.000318 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c8e4e36f-kserve-self-signed-certs\"" Nov 03 20:43:38.001965 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.001927 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj"] Nov 03 20:43:38.063842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.063806 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-dshm\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.063842 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.063846 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-home\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.064104 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.063940 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93407c20-c610-4854-8092-f04763e412ad-tls-certs\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.064104 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.063983 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhj8w\" (UniqueName: \"kubernetes.io/projected/93407c20-c610-4854-8092-f04763e412ad-kube-api-access-fhj8w\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.064104 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.064080 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-tmp-dir\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.064288 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.064112 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-model-cache\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.165216 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.165173 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-dshm\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.165216 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.165219 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-home\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.165492 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.165338 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93407c20-c610-4854-8092-f04763e412ad-tls-certs\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.165492 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.165400 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhj8w\" (UniqueName: \"kubernetes.io/projected/93407c20-c610-4854-8092-f04763e412ad-kube-api-access-fhj8w\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.165492 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.165440 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-tmp-dir\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.165492 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.165483 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-model-cache\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.165820 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.165793 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-tmp-dir\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.165935 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.165901 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-home\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.166003 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.165947 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-model-cache\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.168340 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.168310 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-dshm\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.168525 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.168508 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93407c20-c610-4854-8092-f04763e412ad-tls-certs\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.173330 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.173276 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhj8w\" (UniqueName: \"kubernetes.io/projected/93407c20-c610-4854-8092-f04763e412ad-kube-api-access-fhj8w\") pod \"e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.311521 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.311425 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:38.589630 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:38.589512 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj"] Nov 03 20:43:38.592909 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:43:38.592875 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93407c20_c610_4854_8092_f04763e412ad.slice/crio-9f4edc60810645a13bc3a4d788263d5a6027ddb839c538807375a794e42a3e7f WatchSource:0}: Error finding container 9f4edc60810645a13bc3a4d788263d5a6027ddb839c538807375a794e42a3e7f: Status 404 returned error can't find the container with id 9f4edc60810645a13bc3a4d788263d5a6027ddb839c538807375a794e42a3e7f Nov 03 20:43:39.603245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:39.603149 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" event={"ID":"93407c20-c610-4854-8092-f04763e412ad","Type":"ContainerStarted","Data":"fc101fd3631a83dc277893ef58359a407655abb92aa1c68046ad523f07a28fce"} Nov 03 20:43:39.603245 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:39.603190 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" event={"ID":"93407c20-c610-4854-8092-f04763e412ad","Type":"ContainerStarted","Data":"9f4edc60810645a13bc3a4d788263d5a6027ddb839c538807375a794e42a3e7f"} Nov 03 20:43:39.622800 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:39.622729 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" podStartSLOduration=2.622707706 podStartE2EDuration="2.622707706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:39.619130746 +0000 UTC m=+3849.581509186" watchObservedRunningTime="2026-11-03 20:43:39.622707706 +0000 UTC m=+3849.585086146" Nov 03 20:43:40.313274 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:40.312860 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:48.311788 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:48.311736 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:48.324839 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:48.324806 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:48.698666 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:48.698634 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:43:58.489127 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:58.488717 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8bcwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:43:58.489127 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:43:58.488799 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" podUID="ea1836fd-1444-44ab-831c-165ef7470a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:44:05.056672 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:44:05.056628 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:44:11.442332 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.442295 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g"] Nov 03 20:44:11.446122 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.446100 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" Nov 03 20:44:11.451567 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.451520 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:44:11.451876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.451627 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:44:11.451876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.451679 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:44:11.452064 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.451840 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:44:11.453322 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.453296 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g"] Nov 03 20:44:11.501277 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.501241 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsrpq\" (UniqueName: \"kubernetes.io/projected/f598fc63-845a-43c0-84d1-9189ac70bcae-kube-api-access-vsrpq\") pod \"maas-api-revoke-sub-452b6d7b-nsv5g\" (UID: \"f598fc63-845a-43c0-84d1-9189ac70bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" Nov 03 20:44:11.501508 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.501405 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f598fc63-845a-43c0-84d1-9189ac70bcae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nsv5g\" (UID: \"f598fc63-845a-43c0-84d1-9189ac70bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" Nov 03 20:44:11.501508 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.501474 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f598fc63-845a-43c0-84d1-9189ac70bcae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nsv5g\" (UID: \"f598fc63-845a-43c0-84d1-9189ac70bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" Nov 03 20:44:11.602814 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.602769 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsrpq\" (UniqueName: \"kubernetes.io/projected/f598fc63-845a-43c0-84d1-9189ac70bcae-kube-api-access-vsrpq\") pod \"maas-api-revoke-sub-452b6d7b-nsv5g\" (UID: \"f598fc63-845a-43c0-84d1-9189ac70bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" Nov 03 20:44:11.603013 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.602908 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f598fc63-845a-43c0-84d1-9189ac70bcae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nsv5g\" (UID: \"f598fc63-845a-43c0-84d1-9189ac70bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" Nov 03 20:44:11.603013 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.602941 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f598fc63-845a-43c0-84d1-9189ac70bcae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nsv5g\" (UID: \"f598fc63-845a-43c0-84d1-9189ac70bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" Nov 03 20:44:11.603781 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.603758 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f598fc63-845a-43c0-84d1-9189ac70bcae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nsv5g\" (UID: \"f598fc63-845a-43c0-84d1-9189ac70bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" Nov 03 20:44:11.605382 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.605355 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f598fc63-845a-43c0-84d1-9189ac70bcae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nsv5g\" (UID: \"f598fc63-845a-43c0-84d1-9189ac70bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" Nov 03 20:44:11.614093 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.614065 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsrpq\" (UniqueName: \"kubernetes.io/projected/f598fc63-845a-43c0-84d1-9189ac70bcae-kube-api-access-vsrpq\") pod \"maas-api-revoke-sub-452b6d7b-nsv5g\" (UID: \"f598fc63-845a-43c0-84d1-9189ac70bcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" Nov 03 20:44:11.762177 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:11.762087 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" Nov 03 20:44:12.038918 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:12.038886 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g"] Nov 03 20:44:12.041106 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:44:12.041070 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf598fc63_845a_43c0_84d1_9189ac70bcae.slice/crio-c5bd9268c2c1a77240803652dcab819ebb17b77db992866e729d6fe51496770d WatchSource:0}: Error finding container c5bd9268c2c1a77240803652dcab819ebb17b77db992866e729d6fe51496770d: Status 404 returned error can't find the container with id c5bd9268c2c1a77240803652dcab819ebb17b77db992866e729d6fe51496770d Nov 03 20:44:12.869600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:12.869513 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" event={"ID":"f598fc63-845a-43c0-84d1-9189ac70bcae","Type":"ContainerStarted","Data":"cd4c3edcdb2aadad1959bb56074636ade1885d497bd12beea1d1772a6d9e4da2"} Nov 03 20:44:12.870155 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:12.869621 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" event={"ID":"f598fc63-845a-43c0-84d1-9189ac70bcae","Type":"ContainerStarted","Data":"c5bd9268c2c1a77240803652dcab819ebb17b77db992866e729d6fe51496770d"} Nov 03 20:44:12.886648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:12.886217 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" podStartSLOduration=1.886201379 podStartE2EDuration="1.886201379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:12.885925926 +0000 UTC m=+3882.848304391" watchObservedRunningTime="2026-11-03 20:44:12.886201379 +0000 UTC m=+3882.848579821" Nov 03 20:44:13.940416 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:13.940371 5111 generic.go:356] "Generic (PLEG): container finished" podID="f598fc63-845a-43c0-84d1-9189ac70bcae" containerID="cd4c3edcdb2aadad1959bb56074636ade1885d497bd12beea1d1772a6d9e4da2" exitCode=0 Nov 03 20:44:13.941138 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:13.941111 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" event={"ID":"f598fc63-845a-43c0-84d1-9189ac70bcae","Type":"ContainerDied","Data":"cd4c3edcdb2aadad1959bb56074636ade1885d497bd12beea1d1772a6d9e4da2"} Nov 03 20:44:15.881033 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:15.881001 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" Nov 03 20:44:15.965657 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:15.965629 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" Nov 03 20:44:15.965856 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:15.965670 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g" event={"ID":"f598fc63-845a-43c0-84d1-9189ac70bcae","Type":"ContainerDied","Data":"c5bd9268c2c1a77240803652dcab819ebb17b77db992866e729d6fe51496770d"} Nov 03 20:44:15.965856 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:15.965725 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5bd9268c2c1a77240803652dcab819ebb17b77db992866e729d6fe51496770d" Nov 03 20:44:15.973856 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:15.973836 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f598fc63-845a-43c0-84d1-9189ac70bcae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f598fc63-845a-43c0-84d1-9189ac70bcae-maas-api-openshift-ca\") pod \"f598fc63-845a-43c0-84d1-9189ac70bcae\" (UID: \"f598fc63-845a-43c0-84d1-9189ac70bcae\") " Nov 03 20:44:15.973953 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:15.973927 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f598fc63-845a-43c0-84d1-9189ac70bcae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f598fc63-845a-43c0-84d1-9189ac70bcae-maas-api-xks-ca\") pod \"f598fc63-845a-43c0-84d1-9189ac70bcae\" (UID: \"f598fc63-845a-43c0-84d1-9189ac70bcae\") " Nov 03 20:44:15.973997 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:15.973971 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f598fc63-845a-43c0-84d1-9189ac70bcae-kube-api-access-vsrpq\" (UniqueName: \"kubernetes.io/projected/f598fc63-845a-43c0-84d1-9189ac70bcae-kube-api-access-vsrpq\") pod \"f598fc63-845a-43c0-84d1-9189ac70bcae\" (UID: \"f598fc63-845a-43c0-84d1-9189ac70bcae\") " Nov 03 20:44:15.974260 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:15.974236 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f598fc63-845a-43c0-84d1-9189ac70bcae-maas-api-openshift-ca" pod "f598fc63-845a-43c0-84d1-9189ac70bcae" (UID: "f598fc63-845a-43c0-84d1-9189ac70bcae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:44:15.974445 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:15.974422 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f598fc63-845a-43c0-84d1-9189ac70bcae-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:15.976278 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:15.976248 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f598fc63-845a-43c0-84d1-9189ac70bcae-maas-api-xks-ca" pod "f598fc63-845a-43c0-84d1-9189ac70bcae" (UID: "f598fc63-845a-43c0-84d1-9189ac70bcae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:44:15.976369 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:15.976253 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f598fc63-845a-43c0-84d1-9189ac70bcae-kube-api-access-vsrpq" pod "f598fc63-845a-43c0-84d1-9189ac70bcae" (UID: "f598fc63-845a-43c0-84d1-9189ac70bcae"). InnerVolumeSpecName "kube-api-access-vsrpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:44:16.075850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:16.075769 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f598fc63-845a-43c0-84d1-9189ac70bcae-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:16.075850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:16.075797 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsrpq\" (UniqueName: \"kubernetes.io/projected/f598fc63-845a-43c0-84d1-9189ac70bcae-kube-api-access-vsrpq\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:22.309160 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:22.309111 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj"] Nov 03 20:44:27.652760 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.652710 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf"] Nov 03 20:44:27.654203 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.654153 5111 cpu_manager.go:414] "containerMap: removing container" podUID="f598fc63-845a-43c0-84d1-9189ac70bcae" containerName="revoke-keys" Nov 03 20:44:27.654203 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.654191 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f598fc63-845a-43c0-84d1-9189ac70bcae" containerName="revoke-keys" Nov 03 20:44:27.654465 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.654392 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="f598fc63-845a-43c0-84d1-9189ac70bcae" containerName="revoke-keys" Nov 03 20:44:27.658258 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.658231 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" Nov 03 20:44:27.662931 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.662900 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:44:27.666305 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.666276 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:44:27.668508 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.668164 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf"] Nov 03 20:44:27.683897 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.683846 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:44:27.684103 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.684004 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:44:27.750781 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.750597 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gc4bf\" (UID: \"fb2f72d9-57be-4eb0-8e5a-f031981fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" Nov 03 20:44:27.750781 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.750736 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gc4bf\" (UID: \"fb2f72d9-57be-4eb0-8e5a-f031981fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" Nov 03 20:44:27.751101 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.750815 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9xmp\" (UniqueName: \"kubernetes.io/projected/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-kube-api-access-c9xmp\") pod \"maas-api-revoke-sub-3e31a506-gc4bf\" (UID: \"fb2f72d9-57be-4eb0-8e5a-f031981fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" Nov 03 20:44:27.852546 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.852462 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9xmp\" (UniqueName: \"kubernetes.io/projected/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-kube-api-access-c9xmp\") pod \"maas-api-revoke-sub-3e31a506-gc4bf\" (UID: \"fb2f72d9-57be-4eb0-8e5a-f031981fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" Nov 03 20:44:27.852798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.852651 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gc4bf\" (UID: \"fb2f72d9-57be-4eb0-8e5a-f031981fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" Nov 03 20:44:27.852798 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.852751 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gc4bf\" (UID: \"fb2f72d9-57be-4eb0-8e5a-f031981fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" Nov 03 20:44:27.853631 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.853599 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gc4bf\" (UID: \"fb2f72d9-57be-4eb0-8e5a-f031981fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" Nov 03 20:44:27.856192 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.856162 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gc4bf\" (UID: \"fb2f72d9-57be-4eb0-8e5a-f031981fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" Nov 03 20:44:27.861199 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.861167 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9xmp\" (UniqueName: \"kubernetes.io/projected/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-kube-api-access-c9xmp\") pod \"maas-api-revoke-sub-3e31a506-gc4bf\" (UID: \"fb2f72d9-57be-4eb0-8e5a-f031981fbccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" Nov 03 20:44:27.989118 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:27.989017 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" Nov 03 20:44:28.295486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:28.295447 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf"] Nov 03 20:44:28.297154 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:44:28.297112 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2f72d9_57be_4eb0_8e5a_f031981fbccb.slice/crio-5cce6c93c712519c0999914af0693d40f78d5228f7b8f8aabf170c984d848cc7 WatchSource:0}: Error finding container 5cce6c93c712519c0999914af0693d40f78d5228f7b8f8aabf170c984d848cc7: Status 404 returned error can't find the container with id 5cce6c93c712519c0999914af0693d40f78d5228f7b8f8aabf170c984d848cc7 Nov 03 20:44:29.095257 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:29.095142 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" event={"ID":"fb2f72d9-57be-4eb0-8e5a-f031981fbccb","Type":"ContainerStarted","Data":"64b7a03bd4cbc048602b0f96619c291c14538a5edb9ff3bce5d9c59b30d5e156"} Nov 03 20:44:29.095952 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:29.095909 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" event={"ID":"fb2f72d9-57be-4eb0-8e5a-f031981fbccb","Type":"ContainerStarted","Data":"5cce6c93c712519c0999914af0693d40f78d5228f7b8f8aabf170c984d848cc7"} Nov 03 20:44:30.145498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:30.144947 5111 generic.go:356] "Generic (PLEG): container finished" podID="fb2f72d9-57be-4eb0-8e5a-f031981fbccb" containerID="64b7a03bd4cbc048602b0f96619c291c14538a5edb9ff3bce5d9c59b30d5e156" exitCode=0 Nov 03 20:44:30.145498 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:30.145060 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" event={"ID":"fb2f72d9-57be-4eb0-8e5a-f031981fbccb","Type":"ContainerDied","Data":"64b7a03bd4cbc048602b0f96619c291c14538a5edb9ff3bce5d9c59b30d5e156"} Nov 03 20:44:31.056653 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:31.056616 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:44:31.061088 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:31.061067 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:44:31.062075 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:31.062054 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:44:31.066625 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:31.066604 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:44:31.305078 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:31.305045 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" Nov 03 20:44:31.449886 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:31.449843 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-maas-api-openshift-ca\") pod \"fb2f72d9-57be-4eb0-8e5a-f031981fbccb\" (UID: \"fb2f72d9-57be-4eb0-8e5a-f031981fbccb\") " Nov 03 20:44:31.450118 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:31.449914 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-kube-api-access-c9xmp\" (UniqueName: \"kubernetes.io/projected/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-kube-api-access-c9xmp\") pod \"fb2f72d9-57be-4eb0-8e5a-f031981fbccb\" (UID: \"fb2f72d9-57be-4eb0-8e5a-f031981fbccb\") " Nov 03 20:44:31.450118 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:31.449996 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-maas-api-xks-ca\") pod \"fb2f72d9-57be-4eb0-8e5a-f031981fbccb\" (UID: \"fb2f72d9-57be-4eb0-8e5a-f031981fbccb\") " Nov 03 20:44:31.450311 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:31.450284 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-maas-api-openshift-ca" pod "fb2f72d9-57be-4eb0-8e5a-f031981fbccb" (UID: "fb2f72d9-57be-4eb0-8e5a-f031981fbccb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:44:31.450421 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:31.450403 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:31.452225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:31.452192 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-maas-api-xks-ca" pod "fb2f72d9-57be-4eb0-8e5a-f031981fbccb" (UID: "fb2f72d9-57be-4eb0-8e5a-f031981fbccb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:44:31.452225 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:31.452199 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-kube-api-access-c9xmp" pod "fb2f72d9-57be-4eb0-8e5a-f031981fbccb" (UID: "fb2f72d9-57be-4eb0-8e5a-f031981fbccb"). InnerVolumeSpecName "kube-api-access-c9xmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:44:31.551668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:31.551632 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9xmp\" (UniqueName: \"kubernetes.io/projected/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-kube-api-access-c9xmp\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:31.551668 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:31.551663 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2f72d9-57be-4eb0-8e5a-f031981fbccb-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:32.157911 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:32.157871 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" event={"ID":"fb2f72d9-57be-4eb0-8e5a-f031981fbccb","Type":"ContainerDied","Data":"5cce6c93c712519c0999914af0693d40f78d5228f7b8f8aabf170c984d848cc7"} Nov 03 20:44:32.158099 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:32.157934 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cce6c93c712519c0999914af0693d40f78d5228f7b8f8aabf170c984d848cc7" Nov 03 20:44:32.158099 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:32.157894 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf" Nov 03 20:44:35.057278 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:44:35.057243 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:44:37.330270 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.330197 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" podUID="93407c20-c610-4854-8092-f04763e412ad" containerName="main" containerID="cri-o://fc101fd3631a83dc277893ef58359a407655abb92aa1c68046ad523f07a28fce" gracePeriod=45 Nov 03 20:44:37.597504 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.597428 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:44:37.720903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.720863 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-home\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-home\") pod \"93407c20-c610-4854-8092-f04763e412ad\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " Nov 03 20:44:37.720903 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.720908 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-dshm\") pod \"93407c20-c610-4854-8092-f04763e412ad\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " Nov 03 20:44:37.721190 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.720969 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-model-cache\") pod \"93407c20-c610-4854-8092-f04763e412ad\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " Nov 03 20:44:37.721190 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.721070 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-tmp-dir\") pod \"93407c20-c610-4854-8092-f04763e412ad\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " Nov 03 20:44:37.721190 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.721104 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93407c20-c610-4854-8092-f04763e412ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/93407c20-c610-4854-8092-f04763e412ad-tls-certs\") pod \"93407c20-c610-4854-8092-f04763e412ad\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " Nov 03 20:44:37.721190 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.721179 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93407c20-c610-4854-8092-f04763e412ad-kube-api-access-fhj8w\" (UniqueName: \"kubernetes.io/projected/93407c20-c610-4854-8092-f04763e412ad-kube-api-access-fhj8w\") pod \"93407c20-c610-4854-8092-f04763e412ad\" (UID: \"93407c20-c610-4854-8092-f04763e412ad\") " Nov 03 20:44:37.721408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.721224 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-home" pod "93407c20-c610-4854-8092-f04763e412ad" (UID: "93407c20-c610-4854-8092-f04763e412ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:44:37.721408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.721337 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-model-cache" pod "93407c20-c610-4854-8092-f04763e412ad" (UID: "93407c20-c610-4854-8092-f04763e412ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:44:37.721408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.721359 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-tmp-dir" pod "93407c20-c610-4854-8092-f04763e412ad" (UID: "93407c20-c610-4854-8092-f04763e412ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:44:37.721705 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.721684 5111 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-tmp-dir\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:37.721784 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.721710 5111 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-home\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:37.721784 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.721724 5111 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-model-cache\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:37.723469 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.723441 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93407c20-c610-4854-8092-f04763e412ad-tls-certs" pod "93407c20-c610-4854-8092-f04763e412ad" (UID: "93407c20-c610-4854-8092-f04763e412ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:44:37.723776 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.723746 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-dshm" pod "93407c20-c610-4854-8092-f04763e412ad" (UID: "93407c20-c610-4854-8092-f04763e412ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:44:37.723910 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.723775 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93407c20-c610-4854-8092-f04763e412ad-kube-api-access-fhj8w" pod "93407c20-c610-4854-8092-f04763e412ad" (UID: "93407c20-c610-4854-8092-f04763e412ad"). InnerVolumeSpecName "kube-api-access-fhj8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:44:37.822517 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.822478 5111 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93407c20-c610-4854-8092-f04763e412ad-tls-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:37.822737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.822521 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhj8w\" (UniqueName: \"kubernetes.io/projected/93407c20-c610-4854-8092-f04763e412ad-kube-api-access-fhj8w\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:37.822737 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:37.822571 5111 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93407c20-c610-4854-8092-f04763e412ad-dshm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:38.191861 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:38.191828 5111 generic.go:356] "Generic (PLEG): container finished" podID="93407c20-c610-4854-8092-f04763e412ad" containerID="fc101fd3631a83dc277893ef58359a407655abb92aa1c68046ad523f07a28fce" exitCode=0 Nov 03 20:44:38.192012 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:38.191873 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" Nov 03 20:44:38.192012 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:38.191907 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" event={"ID":"93407c20-c610-4854-8092-f04763e412ad","Type":"ContainerDied","Data":"fc101fd3631a83dc277893ef58359a407655abb92aa1c68046ad523f07a28fce"} Nov 03 20:44:38.192012 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:38.191950 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj" event={"ID":"93407c20-c610-4854-8092-f04763e412ad","Type":"ContainerDied","Data":"9f4edc60810645a13bc3a4d788263d5a6027ddb839c538807375a794e42a3e7f"} Nov 03 20:44:38.192012 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:38.191971 5111 scope.go:122] "RemoveContainer" containerID="fc101fd3631a83dc277893ef58359a407655abb92aa1c68046ad523f07a28fce" Nov 03 20:44:38.201297 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:38.201280 5111 scope.go:122] "RemoveContainer" containerID="fc101fd3631a83dc277893ef58359a407655abb92aa1c68046ad523f07a28fce" Nov 03 20:44:38.201593 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:44:38.201576 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc101fd3631a83dc277893ef58359a407655abb92aa1c68046ad523f07a28fce\": container with ID starting with fc101fd3631a83dc277893ef58359a407655abb92aa1c68046ad523f07a28fce not found: ID does not exist" containerID="fc101fd3631a83dc277893ef58359a407655abb92aa1c68046ad523f07a28fce" Nov 03 20:44:38.201643 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:38.201602 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc101fd3631a83dc277893ef58359a407655abb92aa1c68046ad523f07a28fce"} err="failed to get container status \"fc101fd3631a83dc277893ef58359a407655abb92aa1c68046ad523f07a28fce\": rpc error: code = NotFound desc = could not find container \"fc101fd3631a83dc277893ef58359a407655abb92aa1c68046ad523f07a28fce\": container with ID starting with fc101fd3631a83dc277893ef58359a407655abb92aa1c68046ad523f07a28fce not found: ID does not exist" Nov 03 20:44:38.209057 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:38.209029 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj"] Nov 03 20:44:38.212517 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:38.212493 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c8e4e36f-kserve-7bc469d75c-mgpnj"] Nov 03 20:44:39.057575 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:39.057518 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t"] Nov 03 20:44:39.109559 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:39.098300 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tj64t"] Nov 03 20:44:39.254044 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:39.253893 5111 scope.go:122] "RemoveContainer" containerID="1aa570cf50f30c0d6f8e971f4065b3250d60cc2b371abd62249ffaebdf992fb3" Nov 03 20:44:40.221694 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:40.221653 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93407c20-c610-4854-8092-f04763e412ad" path="/var/lib/kubelet/pods/93407c20-c610-4854-8092-f04763e412ad/volumes" Nov 03 20:44:40.239437 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:40.239401 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6579d9f-39e4-46e0-935c-333653d0615b" path="/var/lib/kubelet/pods/b6579d9f-39e4-46e0-935c-333653d0615b/volumes" Nov 03 20:44:43.484621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:43.483663 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8bcwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:44:43.484621 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:43.483740 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8bcwd" podUID="ea1836fd-1444-44ab-831c-165ef7470a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:44:49.116295 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.116257 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr"] Nov 03 20:44:49.116765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.116730 5111 cpu_manager.go:414] "containerMap: removing container" podUID="fb2f72d9-57be-4eb0-8e5a-f031981fbccb" containerName="revoke-keys" Nov 03 20:44:49.116765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.116742 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb2f72d9-57be-4eb0-8e5a-f031981fbccb" containerName="revoke-keys" Nov 03 20:44:49.116765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.116759 5111 cpu_manager.go:414] "containerMap: removing container" podUID="93407c20-c610-4854-8092-f04763e412ad" containerName="main" Nov 03 20:44:49.116765 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.116764 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93407c20-c610-4854-8092-f04763e412ad" containerName="main" Nov 03 20:44:49.116906 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.116852 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb2f72d9-57be-4eb0-8e5a-f031981fbccb" containerName="revoke-keys" Nov 03 20:44:49.116906 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.116863 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="93407c20-c610-4854-8092-f04763e412ad" containerName="main" Nov 03 20:44:49.121433 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.121412 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" Nov 03 20:44:49.126766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.126726 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:44:49.126766 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.126726 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:44:49.126980 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.126773 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:44:49.126980 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.126775 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:44:49.127646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.127619 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr"] Nov 03 20:44:49.218797 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.218763 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa9d26b-9326-4736-b334-fcf5d6e630f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e32503-h8cgr\" (UID: \"6aa9d26b-9326-4736-b334-fcf5d6e630f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" Nov 03 20:44:49.219002 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.218802 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbh52\" (UniqueName: \"kubernetes.io/projected/6aa9d26b-9326-4736-b334-fcf5d6e630f3-kube-api-access-tbh52\") pod \"maas-api-revoke-sub-85e32503-h8cgr\" (UID: \"6aa9d26b-9326-4736-b334-fcf5d6e630f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" Nov 03 20:44:49.219002 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.218925 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa9d26b-9326-4736-b334-fcf5d6e630f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e32503-h8cgr\" (UID: \"6aa9d26b-9326-4736-b334-fcf5d6e630f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" Nov 03 20:44:49.320080 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.320038 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa9d26b-9326-4736-b334-fcf5d6e630f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e32503-h8cgr\" (UID: \"6aa9d26b-9326-4736-b334-fcf5d6e630f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" Nov 03 20:44:49.320373 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.320089 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbh52\" (UniqueName: \"kubernetes.io/projected/6aa9d26b-9326-4736-b334-fcf5d6e630f3-kube-api-access-tbh52\") pod \"maas-api-revoke-sub-85e32503-h8cgr\" (UID: \"6aa9d26b-9326-4736-b334-fcf5d6e630f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" Nov 03 20:44:49.320373 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.320172 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa9d26b-9326-4736-b334-fcf5d6e630f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e32503-h8cgr\" (UID: \"6aa9d26b-9326-4736-b334-fcf5d6e630f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" Nov 03 20:44:49.320816 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.320791 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa9d26b-9326-4736-b334-fcf5d6e630f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e32503-h8cgr\" (UID: \"6aa9d26b-9326-4736-b334-fcf5d6e630f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" Nov 03 20:44:49.322794 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.322768 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa9d26b-9326-4736-b334-fcf5d6e630f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e32503-h8cgr\" (UID: \"6aa9d26b-9326-4736-b334-fcf5d6e630f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" Nov 03 20:44:49.328887 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.328861 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbh52\" (UniqueName: \"kubernetes.io/projected/6aa9d26b-9326-4736-b334-fcf5d6e630f3-kube-api-access-tbh52\") pod \"maas-api-revoke-sub-85e32503-h8cgr\" (UID: \"6aa9d26b-9326-4736-b334-fcf5d6e630f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" Nov 03 20:44:49.437178 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.437138 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" Nov 03 20:44:49.712100 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:49.711992 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr"] Nov 03 20:44:49.714747 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:44:49.714717 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa9d26b_9326_4736_b334_fcf5d6e630f3.slice/crio-5e26f99407ce7b71c7785b3a1858777019f5e39ccd1676f78e502b13c60c8d62 WatchSource:0}: Error finding container 5e26f99407ce7b71c7785b3a1858777019f5e39ccd1676f78e502b13c60c8d62: Status 404 returned error can't find the container with id 5e26f99407ce7b71c7785b3a1858777019f5e39ccd1676f78e502b13c60c8d62 Nov 03 20:44:50.413832 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:50.413791 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" event={"ID":"6aa9d26b-9326-4736-b334-fcf5d6e630f3","Type":"ContainerStarted","Data":"85162e6d29e44f7077098d36e875061105dd9ed473d3af894c43648e8b9e5e5e"} Nov 03 20:44:50.414265 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:50.413859 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" event={"ID":"6aa9d26b-9326-4736-b334-fcf5d6e630f3","Type":"ContainerStarted","Data":"5e26f99407ce7b71c7785b3a1858777019f5e39ccd1676f78e502b13c60c8d62"} Nov 03 20:44:50.430242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:50.430188 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" podStartSLOduration=1.430172904 podStartE2EDuration="1.430172904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:50.428151247 +0000 UTC m=+3920.390529685" watchObservedRunningTime="2026-11-03 20:44:50.430172904 +0000 UTC m=+3920.392551364" Nov 03 20:44:51.421134 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:51.421096 5111 generic.go:356] "Generic (PLEG): container finished" podID="6aa9d26b-9326-4736-b334-fcf5d6e630f3" containerID="85162e6d29e44f7077098d36e875061105dd9ed473d3af894c43648e8b9e5e5e" exitCode=0 Nov 03 20:44:51.421603 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:51.421183 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" event={"ID":"6aa9d26b-9326-4736-b334-fcf5d6e630f3","Type":"ContainerDied","Data":"85162e6d29e44f7077098d36e875061105dd9ed473d3af894c43648e8b9e5e5e"} Nov 03 20:44:52.586894 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:52.586867 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" Nov 03 20:44:52.655666 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:52.655628 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa9d26b-9326-4736-b334-fcf5d6e630f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa9d26b-9326-4736-b334-fcf5d6e630f3-maas-api-openshift-ca\") pod \"6aa9d26b-9326-4736-b334-fcf5d6e630f3\" (UID: \"6aa9d26b-9326-4736-b334-fcf5d6e630f3\") " Nov 03 20:44:52.655858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:52.655697 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa9d26b-9326-4736-b334-fcf5d6e630f3-kube-api-access-tbh52\" (UniqueName: \"kubernetes.io/projected/6aa9d26b-9326-4736-b334-fcf5d6e630f3-kube-api-access-tbh52\") pod \"6aa9d26b-9326-4736-b334-fcf5d6e630f3\" (UID: \"6aa9d26b-9326-4736-b334-fcf5d6e630f3\") " Nov 03 20:44:52.655858 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:52.655725 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aa9d26b-9326-4736-b334-fcf5d6e630f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa9d26b-9326-4736-b334-fcf5d6e630f3-maas-api-xks-ca\") pod \"6aa9d26b-9326-4736-b334-fcf5d6e630f3\" (UID: \"6aa9d26b-9326-4736-b334-fcf5d6e630f3\") " Nov 03 20:44:52.656030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:52.656006 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa9d26b-9326-4736-b334-fcf5d6e630f3-maas-api-openshift-ca" pod "6aa9d26b-9326-4736-b334-fcf5d6e630f3" (UID: "6aa9d26b-9326-4736-b334-fcf5d6e630f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:44:52.656227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:52.656202 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa9d26b-9326-4736-b334-fcf5d6e630f3-maas-api-openshift-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:52.657902 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:52.657872 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa9d26b-9326-4736-b334-fcf5d6e630f3-maas-api-xks-ca" pod "6aa9d26b-9326-4736-b334-fcf5d6e630f3" (UID: "6aa9d26b-9326-4736-b334-fcf5d6e630f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:44:52.658025 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:52.658002 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa9d26b-9326-4736-b334-fcf5d6e630f3-kube-api-access-tbh52" pod "6aa9d26b-9326-4736-b334-fcf5d6e630f3" (UID: "6aa9d26b-9326-4736-b334-fcf5d6e630f3"). InnerVolumeSpecName "kube-api-access-tbh52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:44:52.757218 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:52.757130 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbh52\" (UniqueName: \"kubernetes.io/projected/6aa9d26b-9326-4736-b334-fcf5d6e630f3-kube-api-access-tbh52\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:52.757218 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:52.757163 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa9d26b-9326-4736-b334-fcf5d6e630f3-maas-api-xks-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:53.433182 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:53.433148 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" Nov 03 20:44:53.433182 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:53.433159 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr" event={"ID":"6aa9d26b-9326-4736-b334-fcf5d6e630f3","Type":"ContainerDied","Data":"5e26f99407ce7b71c7785b3a1858777019f5e39ccd1676f78e502b13c60c8d62"} Nov 03 20:44:53.433408 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:44:53.433215 5111 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e26f99407ce7b71c7785b3a1858777019f5e39ccd1676f78e502b13c60c8d62" Nov 03 20:45:00.140156 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.140117 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n"] Nov 03 20:45:00.140707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.140629 5111 cpu_manager.go:414] "containerMap: removing container" podUID="6aa9d26b-9326-4736-b334-fcf5d6e630f3" containerName="revoke-keys" Nov 03 20:45:00.140707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.140645 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa9d26b-9326-4736-b334-fcf5d6e630f3" containerName="revoke-keys" Nov 03 20:45:00.140819 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.140756 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa9d26b-9326-4736-b334-fcf5d6e630f3" containerName="revoke-keys" Nov 03 20:45:00.144372 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.144338 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" Nov 03 20:45:00.150650 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.150254 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\"" Nov 03 20:45:00.152202 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.152168 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n"] Nov 03 20:45:00.166191 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.166145 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:45:00.166397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.166227 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:45:00.234444 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.234405 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcccm\" (UniqueName: \"kubernetes.io/projected/3bf22ba3-3e73-4cc9-acd6-467414ab7896-kube-api-access-dcccm\") pod \"maas-api-key-cleanup-29895645-ffs5n\" (UID: \"3bf22ba3-3e73-4cc9-acd6-467414ab7896\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" Nov 03 20:45:00.234650 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.234479 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf22ba3-3e73-4cc9-acd6-467414ab7896-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-ffs5n\" (UID: \"3bf22ba3-3e73-4cc9-acd6-467414ab7896\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" Nov 03 20:45:00.336216 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.336162 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf22ba3-3e73-4cc9-acd6-467414ab7896-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-ffs5n\" (UID: \"3bf22ba3-3e73-4cc9-acd6-467414ab7896\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" Nov 03 20:45:00.337781 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.336522 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcccm\" (UniqueName: \"kubernetes.io/projected/3bf22ba3-3e73-4cc9-acd6-467414ab7896-kube-api-access-dcccm\") pod \"maas-api-key-cleanup-29895645-ffs5n\" (UID: \"3bf22ba3-3e73-4cc9-acd6-467414ab7896\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" Nov 03 20:45:00.337781 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.337732 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf22ba3-3e73-4cc9-acd6-467414ab7896-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-ffs5n\" (UID: \"3bf22ba3-3e73-4cc9-acd6-467414ab7896\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" Nov 03 20:45:00.347089 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.347035 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcccm\" (UniqueName: \"kubernetes.io/projected/3bf22ba3-3e73-4cc9-acd6-467414ab7896-kube-api-access-dcccm\") pod \"maas-api-key-cleanup-29895645-ffs5n\" (UID: \"3bf22ba3-3e73-4cc9-acd6-467414ab7896\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" Nov 03 20:45:00.460673 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.460585 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" Nov 03 20:45:00.737059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:00.737026 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n"] Nov 03 20:45:00.739942 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:45:00.739909 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf22ba3_3e73_4cc9_acd6_467414ab7896.slice/crio-73d8d4b751a46c38d8588fcef7fdd2bdb525a7bf6cfff486be05498792277136 WatchSource:0}: Error finding container 73d8d4b751a46c38d8588fcef7fdd2bdb525a7bf6cfff486be05498792277136: Status 404 returned error can't find the container with id 73d8d4b751a46c38d8588fcef7fdd2bdb525a7bf6cfff486be05498792277136 Nov 03 20:45:01.340523 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.340435 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh"] Nov 03 20:45:01.344223 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.344203 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:01.352919 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.352892 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7bzpm\"" Nov 03 20:45:01.353648 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.353623 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 20:45:01.353791 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.353638 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:45:01.355926 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.355898 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh"] Nov 03 20:45:01.447789 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.447750 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f1edfb-f711-475f-b517-0800f594d261-metrics-certs\") pod \"maas-api-5cb65579d5-xx5vh\" (UID: \"28f1edfb-f711-475f-b517-0800f594d261\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:01.447966 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.447816 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f1edfb-f711-475f-b517-0800f594d261-maas-api-tls\") pod \"maas-api-5cb65579d5-xx5vh\" (UID: \"28f1edfb-f711-475f-b517-0800f594d261\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:01.447966 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.447890 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f1edfb-f711-475f-b517-0800f594d261-openshift-service-ca\") pod \"maas-api-5cb65579d5-xx5vh\" (UID: \"28f1edfb-f711-475f-b517-0800f594d261\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:01.447966 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.447956 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg5nc\" (UniqueName: \"kubernetes.io/projected/28f1edfb-f711-475f-b517-0800f594d261-kube-api-access-tg5nc\") pod \"maas-api-5cb65579d5-xx5vh\" (UID: \"28f1edfb-f711-475f-b517-0800f594d261\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:01.486585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.486514 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" event={"ID":"3bf22ba3-3e73-4cc9-acd6-467414ab7896","Type":"ContainerStarted","Data":"f3eb671725ff18213fed879969cc1e48fe313f2c7ae07ed9ea1cd6422a2e9915"} Nov 03 20:45:01.486780 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.486607 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" event={"ID":"3bf22ba3-3e73-4cc9-acd6-467414ab7896","Type":"ContainerStarted","Data":"73d8d4b751a46c38d8588fcef7fdd2bdb525a7bf6cfff486be05498792277136"} Nov 03 20:45:01.501928 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.501872 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" podStartSLOduration=1.501856497 podStartE2EDuration="1.501856497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:01.500636161 +0000 UTC m=+3931.463014601" watchObservedRunningTime="2026-11-03 20:45:01.501856497 +0000 UTC m=+3931.464234935" Nov 03 20:45:01.549341 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.549302 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f1edfb-f711-475f-b517-0800f594d261-maas-api-tls\") pod \"maas-api-5cb65579d5-xx5vh\" (UID: \"28f1edfb-f711-475f-b517-0800f594d261\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:01.549574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.549346 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f1edfb-f711-475f-b517-0800f594d261-openshift-service-ca\") pod \"maas-api-5cb65579d5-xx5vh\" (UID: \"28f1edfb-f711-475f-b517-0800f594d261\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:01.549574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.549408 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg5nc\" (UniqueName: \"kubernetes.io/projected/28f1edfb-f711-475f-b517-0800f594d261-kube-api-access-tg5nc\") pod \"maas-api-5cb65579d5-xx5vh\" (UID: \"28f1edfb-f711-475f-b517-0800f594d261\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:01.549574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.549551 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f1edfb-f711-475f-b517-0800f594d261-metrics-certs\") pod \"maas-api-5cb65579d5-xx5vh\" (UID: \"28f1edfb-f711-475f-b517-0800f594d261\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:01.550101 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.550071 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f1edfb-f711-475f-b517-0800f594d261-openshift-service-ca\") pod \"maas-api-5cb65579d5-xx5vh\" (UID: \"28f1edfb-f711-475f-b517-0800f594d261\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:01.552106 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.552057 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f1edfb-f711-475f-b517-0800f594d261-metrics-certs\") pod \"maas-api-5cb65579d5-xx5vh\" (UID: \"28f1edfb-f711-475f-b517-0800f594d261\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:01.552211 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.552119 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f1edfb-f711-475f-b517-0800f594d261-maas-api-tls\") pod \"maas-api-5cb65579d5-xx5vh\" (UID: \"28f1edfb-f711-475f-b517-0800f594d261\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:01.561126 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.561099 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg5nc\" (UniqueName: \"kubernetes.io/projected/28f1edfb-f711-475f-b517-0800f594d261-kube-api-access-tg5nc\") pod \"maas-api-5cb65579d5-xx5vh\" (UID: \"28f1edfb-f711-475f-b517-0800f594d261\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:01.661029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.660996 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:01.938712 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.938685 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh"] Nov 03 20:45:01.940742 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:45:01.940708 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f1edfb_f711_475f_b517_0800f594d261.slice/crio-8f3fdf162fea2d0001234ed46c1214ecffd5a935b6889df7ec0d7fa29b42afb7 WatchSource:0}: Error finding container 8f3fdf162fea2d0001234ed46c1214ecffd5a935b6889df7ec0d7fa29b42afb7: Status 404 returned error can't find the container with id 8f3fdf162fea2d0001234ed46c1214ecffd5a935b6889df7ec0d7fa29b42afb7 Nov 03 20:45:01.942418 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:01.942396 5111 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:45:02.494214 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:02.494157 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" event={"ID":"28f1edfb-f711-475f-b517-0800f594d261","Type":"ContainerStarted","Data":"e49cb3a807f4c2fce389780f726ddb271fe46603b21136959bd5720a24fbe9ae"} Nov 03 20:45:02.494690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:02.494343 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:02.494690 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:02.494374 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" event={"ID":"28f1edfb-f711-475f-b517-0800f594d261","Type":"ContainerStarted","Data":"8f3fdf162fea2d0001234ed46c1214ecffd5a935b6889df7ec0d7fa29b42afb7"} Nov 03 20:45:02.496096 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:02.496068 5111 generic.go:356] "Generic (PLEG): container finished" podID="3bf22ba3-3e73-4cc9-acd6-467414ab7896" containerID="f3eb671725ff18213fed879969cc1e48fe313f2c7ae07ed9ea1cd6422a2e9915" exitCode=60 Nov 03 20:45:02.496277 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:02.496144 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" event={"ID":"3bf22ba3-3e73-4cc9-acd6-467414ab7896","Type":"ContainerDied","Data":"f3eb671725ff18213fed879969cc1e48fe313f2c7ae07ed9ea1cd6422a2e9915"} Nov 03 20:45:02.496398 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:02.496378 5111 scope.go:122] "RemoveContainer" containerID="f3eb671725ff18213fed879969cc1e48fe313f2c7ae07ed9ea1cd6422a2e9915" Nov 03 20:45:02.512880 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:02.512814 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" podStartSLOduration=1.033876336 podStartE2EDuration="1.512797023s" totalImagesPullingTime="478.920687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:02.511842268 +0000 UTC m=+3932.474220706" watchObservedRunningTime="2026-11-03 20:45:02.512797023 +0000 UTC m=+3932.475175464" Nov 03 20:45:03.503507 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:03.503474 5111 generic.go:356] "Generic (PLEG): container finished" podID="3bf22ba3-3e73-4cc9-acd6-467414ab7896" containerID="d109d4d307444a9129d7c0be6864d961da533a8b18704124a92be11e2a5bcdae" exitCode=60 Nov 03 20:45:03.503992 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:03.503560 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" event={"ID":"3bf22ba3-3e73-4cc9-acd6-467414ab7896","Type":"ContainerDied","Data":"d109d4d307444a9129d7c0be6864d961da533a8b18704124a92be11e2a5bcdae"} Nov 03 20:45:03.503992 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:03.503617 5111 scope.go:122] "RemoveContainer" containerID="f3eb671725ff18213fed879969cc1e48fe313f2c7ae07ed9ea1cd6422a2e9915" Nov 03 20:45:03.503992 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:03.503868 5111 scope.go:122] "RemoveContainer" containerID="d109d4d307444a9129d7c0be6864d961da533a8b18704124a92be11e2a5bcdae" Nov 03 20:45:03.504193 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:45:03.504147 5111 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-ffs5n_odh-ai-gateway-infra(3bf22ba3-3e73-4cc9-acd6-467414ab7896)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" podUID="3bf22ba3-3e73-4cc9-acd6-467414ab7896" Nov 03 20:45:04.510679 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:04.510648 5111 scope.go:122] "RemoveContainer" containerID="d109d4d307444a9129d7c0be6864d961da533a8b18704124a92be11e2a5bcdae" Nov 03 20:45:04.511160 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:45:04.510897 5111 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-ffs5n_odh-ai-gateway-infra(3bf22ba3-3e73-4cc9-acd6-467414ab7896)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" podUID="3bf22ba3-3e73-4cc9-acd6-467414ab7896" Nov 03 20:45:05.060510 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:45:05.060472 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:45:08.509731 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.509694 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:08.636884 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.636847 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4"] Nov 03 20:45:08.640783 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.640760 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.649571 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.649510 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-364642e1-openshift-default-dockercfg-f7lt2\"" Nov 03 20:45:08.654643 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.654609 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4"] Nov 03 20:45:08.829767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.829682 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-data\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.829767 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.829742 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-workload-socket\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.829993 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.829840 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-token\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.829993 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.829890 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-credential-socket\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.829993 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.829951 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-podinfo\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.829993 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.829990 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2997k\" (UniqueName: \"kubernetes.io/projected/b9cdc815-0afe-469b-b8da-19b5c92cfd79-kube-api-access-2997k\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.830176 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.830046 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-envoy\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.830176 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.830081 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-workload-certs\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.830176 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.830096 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istiod-ca-cert\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.872989 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.872952 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-876c878b5-lxzrv"] Nov 03 20:45:08.876960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.876934 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" Nov 03 20:45:08.887021 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.886993 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-876c878b5-lxzrv"] Nov 03 20:45:08.931324 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.931288 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-data\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.931574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.931339 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-workload-socket\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.931574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.931457 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-token\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.931574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.931500 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-credential-socket\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.931574 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.931528 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-podinfo\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.931850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.931613 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2997k\" (UniqueName: \"kubernetes.io/projected/b9cdc815-0afe-469b-b8da-19b5c92cfd79-kube-api-access-2997k\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.931850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.931668 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-envoy\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.931850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.931702 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-workload-certs\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.931850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.931726 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istiod-ca-cert\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.931850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.931771 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-workload-socket\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.931850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.931802 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-data\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.931850 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.931837 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-credential-socket\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.932241 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.932161 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-workload-certs\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.932495 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.932474 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istiod-ca-cert\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.934042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.934023 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-envoy\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.934217 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.934201 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-podinfo\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.940069 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.940045 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-token\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.940365 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.940344 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2997k\" (UniqueName: \"kubernetes.io/projected/b9cdc815-0afe-469b-b8da-19b5c92cfd79-kube-api-access-2997k\") pod \"e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:08.960799 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:08.960697 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:09.033163 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.033098 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84j76\" (UniqueName: \"kubernetes.io/projected/d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3-kube-api-access-84j76\") pod \"maas-api-876c878b5-lxzrv\" (UID: \"d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" Nov 03 20:45:09.033163 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.033146 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3-metrics-certs\") pod \"maas-api-876c878b5-lxzrv\" (UID: \"d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" Nov 03 20:45:09.033414 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.033240 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3-openshift-service-ca\") pod \"maas-api-876c878b5-lxzrv\" (UID: \"d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" Nov 03 20:45:09.033414 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.033282 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3-maas-api-tls\") pod \"maas-api-876c878b5-lxzrv\" (UID: \"d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" Nov 03 20:45:09.134234 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.134188 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3-openshift-service-ca\") pod \"maas-api-876c878b5-lxzrv\" (UID: \"d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" Nov 03 20:45:09.134475 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.134263 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3-maas-api-tls\") pod \"maas-api-876c878b5-lxzrv\" (UID: \"d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" Nov 03 20:45:09.134475 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.134406 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84j76\" (UniqueName: \"kubernetes.io/projected/d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3-kube-api-access-84j76\") pod \"maas-api-876c878b5-lxzrv\" (UID: \"d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" Nov 03 20:45:09.134475 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.134434 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3-metrics-certs\") pod \"maas-api-876c878b5-lxzrv\" (UID: \"d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" Nov 03 20:45:09.135130 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.135096 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3-openshift-service-ca\") pod \"maas-api-876c878b5-lxzrv\" (UID: \"d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" Nov 03 20:45:09.137307 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.137263 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3-metrics-certs\") pod \"maas-api-876c878b5-lxzrv\" (UID: \"d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" Nov 03 20:45:09.137307 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.137287 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3-maas-api-tls\") pod \"maas-api-876c878b5-lxzrv\" (UID: \"d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" Nov 03 20:45:09.144092 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.144063 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84j76\" (UniqueName: \"kubernetes.io/projected/d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3-kube-api-access-84j76\") pod \"maas-api-876c878b5-lxzrv\" (UID: \"d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" Nov 03 20:45:09.193200 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.193162 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" Nov 03 20:45:09.253514 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.253477 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4"] Nov 03 20:45:09.255424 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:45:09.255382 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9cdc815_0afe_469b_b8da_19b5c92cfd79.slice/crio-8eda643b59d8efe4e1129ecb4db504e9387fa08e197cca56c6d6558a6121e393 WatchSource:0}: Error finding container 8eda643b59d8efe4e1129ecb4db504e9387fa08e197cca56c6d6558a6121e393: Status 404 returned error can't find the container with id 8eda643b59d8efe4e1129ecb4db504e9387fa08e197cca56c6d6558a6121e393 Nov 03 20:45:09.260861 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.260810 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:45:09.260994 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.260923 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:45:09.260994 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.260968 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:45:09.473909 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.473879 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-876c878b5-lxzrv"] Nov 03 20:45:09.476701 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:45:09.476669 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8167fb5_7d3e_479c_a2e8_bd0ae3498ba3.slice/crio-373eaf28effc83c49ca47c941bf3d131a042b3285641a7e27d4b8e6deacbab94 WatchSource:0}: Error finding container 373eaf28effc83c49ca47c941bf3d131a042b3285641a7e27d4b8e6deacbab94: Status 404 returned error can't find the container with id 373eaf28effc83c49ca47c941bf3d131a042b3285641a7e27d4b8e6deacbab94 Nov 03 20:45:09.543003 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.542963 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" event={"ID":"d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3","Type":"ContainerStarted","Data":"373eaf28effc83c49ca47c941bf3d131a042b3285641a7e27d4b8e6deacbab94"} Nov 03 20:45:09.544759 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.544729 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" event={"ID":"b9cdc815-0afe-469b-b8da-19b5c92cfd79","Type":"ContainerStarted","Data":"d2d47258251bd352911dfcc298dba7456bb06e19a0a9e5e0cf932649d8eb1efd"} Nov 03 20:45:09.544931 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.544784 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" event={"ID":"b9cdc815-0afe-469b-b8da-19b5c92cfd79","Type":"ContainerStarted","Data":"8eda643b59d8efe4e1129ecb4db504e9387fa08e197cca56c6d6558a6121e393"} Nov 03 20:45:09.568499 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.568440 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" podStartSLOduration=1.5684252810000001 podStartE2EDuration="1.568425281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:09.567523233 +0000 UTC m=+3939.529901673" watchObservedRunningTime="2026-11-03 20:45:09.568425281 +0000 UTC m=+3939.530803713" Nov 03 20:45:09.961634 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:09.961601 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:10.552042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:10.551994 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" event={"ID":"d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3","Type":"ContainerStarted","Data":"840dd5e2a753df57f1b372260cfc67609668a944ae488969875560f97b0d78ea"} Nov 03 20:45:10.552584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:10.552216 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" Nov 03 20:45:10.593343 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:10.593260 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" podStartSLOduration=1.9843309040000001 podStartE2EDuration="2.593245236s" totalImagesPullingTime="608.914332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:10.592062273 +0000 UTC m=+3940.554440711" watchObservedRunningTime="2026-11-03 20:45:10.593245236 +0000 UTC m=+3940.555623675" Nov 03 20:45:10.966936 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:10.966897 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:11.558979 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:11.558942 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:11.559925 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:11.559902 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:16.573519 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:16.573485 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-876c878b5-lxzrv" Nov 03 20:45:16.622006 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:16.621961 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh"] Nov 03 20:45:16.622224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:16.622199 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" podUID="28f1edfb-f711-475f-b517-0800f594d261" containerName="maas-api" containerID="cri-o://e49cb3a807f4c2fce389780f726ddb271fe46603b21136959bd5720a24fbe9ae" gracePeriod=30 Nov 03 20:45:16.933473 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:16.933443 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:17.041602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.041568 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28f1edfb-f711-475f-b517-0800f594d261-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f1edfb-f711-475f-b517-0800f594d261-metrics-certs\") pod \"28f1edfb-f711-475f-b517-0800f594d261\" (UID: \"28f1edfb-f711-475f-b517-0800f594d261\") " Nov 03 20:45:17.041787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.041649 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28f1edfb-f711-475f-b517-0800f594d261-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f1edfb-f711-475f-b517-0800f594d261-maas-api-tls\") pod \"28f1edfb-f711-475f-b517-0800f594d261\" (UID: \"28f1edfb-f711-475f-b517-0800f594d261\") " Nov 03 20:45:17.041787 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.041703 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f1edfb-f711-475f-b517-0800f594d261-kube-api-access-tg5nc\" (UniqueName: \"kubernetes.io/projected/28f1edfb-f711-475f-b517-0800f594d261-kube-api-access-tg5nc\") pod \"28f1edfb-f711-475f-b517-0800f594d261\" (UID: \"28f1edfb-f711-475f-b517-0800f594d261\") " Nov 03 20:45:17.041874 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.041864 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f1edfb-f711-475f-b517-0800f594d261-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f1edfb-f711-475f-b517-0800f594d261-openshift-service-ca\") pod \"28f1edfb-f711-475f-b517-0800f594d261\" (UID: \"28f1edfb-f711-475f-b517-0800f594d261\") " Nov 03 20:45:17.042228 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.042201 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f1edfb-f711-475f-b517-0800f594d261-openshift-service-ca" pod "28f1edfb-f711-475f-b517-0800f594d261" (UID: "28f1edfb-f711-475f-b517-0800f594d261"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:17.043853 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.043826 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f1edfb-f711-475f-b517-0800f594d261-metrics-certs" pod "28f1edfb-f711-475f-b517-0800f594d261" (UID: "28f1edfb-f711-475f-b517-0800f594d261"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:45:17.043965 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.043900 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f1edfb-f711-475f-b517-0800f594d261-maas-api-tls" pod "28f1edfb-f711-475f-b517-0800f594d261" (UID: "28f1edfb-f711-475f-b517-0800f594d261"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:45:17.044014 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.043958 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f1edfb-f711-475f-b517-0800f594d261-kube-api-access-tg5nc" pod "28f1edfb-f711-475f-b517-0800f594d261" (UID: "28f1edfb-f711-475f-b517-0800f594d261"). InnerVolumeSpecName "kube-api-access-tg5nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:17.143700 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.143666 5111 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f1edfb-f711-475f-b517-0800f594d261-openshift-service-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:17.143700 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.143701 5111 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f1edfb-f711-475f-b517-0800f594d261-metrics-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:17.143921 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.143713 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f1edfb-f711-475f-b517-0800f594d261-maas-api-tls\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:17.143921 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.143722 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg5nc\" (UniqueName: \"kubernetes.io/projected/28f1edfb-f711-475f-b517-0800f594d261-kube-api-access-tg5nc\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:17.175960 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.175925 5111 scope.go:122] "RemoveContainer" containerID="d109d4d307444a9129d7c0be6864d961da533a8b18704124a92be11e2a5bcdae" Nov 03 20:45:17.604091 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.604053 5111 generic.go:356] "Generic (PLEG): container finished" podID="28f1edfb-f711-475f-b517-0800f594d261" containerID="e49cb3a807f4c2fce389780f726ddb271fe46603b21136959bd5720a24fbe9ae" exitCode=0 Nov 03 20:45:17.604636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.604101 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" Nov 03 20:45:17.604636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.604131 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" event={"ID":"28f1edfb-f711-475f-b517-0800f594d261","Type":"ContainerDied","Data":"e49cb3a807f4c2fce389780f726ddb271fe46603b21136959bd5720a24fbe9ae"} Nov 03 20:45:17.604636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.604183 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh" event={"ID":"28f1edfb-f711-475f-b517-0800f594d261","Type":"ContainerDied","Data":"8f3fdf162fea2d0001234ed46c1214ecffd5a935b6889df7ec0d7fa29b42afb7"} Nov 03 20:45:17.604636 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.604205 5111 scope.go:122] "RemoveContainer" containerID="e49cb3a807f4c2fce389780f726ddb271fe46603b21136959bd5720a24fbe9ae" Nov 03 20:45:17.606456 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.606429 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" event={"ID":"3bf22ba3-3e73-4cc9-acd6-467414ab7896","Type":"ContainerStarted","Data":"3c1ef500c11902e34b8f126210dc0cd1dd8752f022428d55eb45dac64bf75c76"} Nov 03 20:45:17.614210 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.614193 5111 scope.go:122] "RemoveContainer" containerID="e49cb3a807f4c2fce389780f726ddb271fe46603b21136959bd5720a24fbe9ae" Nov 03 20:45:17.614477 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:45:17.614458 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49cb3a807f4c2fce389780f726ddb271fe46603b21136959bd5720a24fbe9ae\": container with ID starting with e49cb3a807f4c2fce389780f726ddb271fe46603b21136959bd5720a24fbe9ae not found: ID does not exist" containerID="e49cb3a807f4c2fce389780f726ddb271fe46603b21136959bd5720a24fbe9ae" Nov 03 20:45:17.614566 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.614491 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49cb3a807f4c2fce389780f726ddb271fe46603b21136959bd5720a24fbe9ae"} err="failed to get container status \"e49cb3a807f4c2fce389780f726ddb271fe46603b21136959bd5720a24fbe9ae\": rpc error: code = NotFound desc = could not find container \"e49cb3a807f4c2fce389780f726ddb271fe46603b21136959bd5720a24fbe9ae\": container with ID starting with e49cb3a807f4c2fce389780f726ddb271fe46603b21136959bd5720a24fbe9ae not found: ID does not exist" Nov 03 20:45:17.638778 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.638688 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh"] Nov 03 20:45:17.642266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:17.642235 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb65579d5-xx5vh"] Nov 03 20:45:18.181912 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:18.181884 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f1edfb-f711-475f-b517-0800f594d261" path="/var/lib/kubelet/pods/28f1edfb-f711-475f-b517-0800f594d261/volumes" Nov 03 20:45:18.615356 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:18.615270 5111 generic.go:356] "Generic (PLEG): container finished" podID="3bf22ba3-3e73-4cc9-acd6-467414ab7896" containerID="3c1ef500c11902e34b8f126210dc0cd1dd8752f022428d55eb45dac64bf75c76" exitCode=60 Nov 03 20:45:18.615356 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:18.615335 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" event={"ID":"3bf22ba3-3e73-4cc9-acd6-467414ab7896","Type":"ContainerDied","Data":"3c1ef500c11902e34b8f126210dc0cd1dd8752f022428d55eb45dac64bf75c76"} Nov 03 20:45:18.615891 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:18.615398 5111 scope.go:122] "RemoveContainer" containerID="d109d4d307444a9129d7c0be6864d961da533a8b18704124a92be11e2a5bcdae" Nov 03 20:45:18.615891 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:18.615817 5111 scope.go:122] "RemoveContainer" containerID="3c1ef500c11902e34b8f126210dc0cd1dd8752f022428d55eb45dac64bf75c76" Nov 03 20:45:18.616142 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:45:18.616116 5111 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-ffs5n_odh-ai-gateway-infra(3bf22ba3-3e73-4cc9-acd6-467414ab7896)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" podUID="3bf22ba3-3e73-4cc9-acd6-467414ab7896" Nov 03 20:45:18.632140 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:18.632114 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n"] Nov 03 20:45:19.807915 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:19.807885 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" Nov 03 20:45:19.893583 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:19.893472 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf22ba3-3e73-4cc9-acd6-467414ab7896-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf22ba3-3e73-4cc9-acd6-467414ab7896-maas-api-service-ca\") pod \"3bf22ba3-3e73-4cc9-acd6-467414ab7896\" (UID: \"3bf22ba3-3e73-4cc9-acd6-467414ab7896\") " Nov 03 20:45:19.894269 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:19.894248 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf22ba3-3e73-4cc9-acd6-467414ab7896-kube-api-access-dcccm\" (UniqueName: \"kubernetes.io/projected/3bf22ba3-3e73-4cc9-acd6-467414ab7896-kube-api-access-dcccm\") pod \"3bf22ba3-3e73-4cc9-acd6-467414ab7896\" (UID: \"3bf22ba3-3e73-4cc9-acd6-467414ab7896\") " Nov 03 20:45:19.895131 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:19.894117 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf22ba3-3e73-4cc9-acd6-467414ab7896-maas-api-service-ca" pod "3bf22ba3-3e73-4cc9-acd6-467414ab7896" (UID: "3bf22ba3-3e73-4cc9-acd6-467414ab7896"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:19.896623 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:19.896599 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf22ba3-3e73-4cc9-acd6-467414ab7896-kube-api-access-dcccm" pod "3bf22ba3-3e73-4cc9-acd6-467414ab7896" (UID: "3bf22ba3-3e73-4cc9-acd6-467414ab7896"). InnerVolumeSpecName "kube-api-access-dcccm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:19.996406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:19.996357 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcccm\" (UniqueName: \"kubernetes.io/projected/3bf22ba3-3e73-4cc9-acd6-467414ab7896-kube-api-access-dcccm\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:19.996406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:19.996401 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf22ba3-3e73-4cc9-acd6-467414ab7896-maas-api-service-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:20.629707 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:20.629679 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" Nov 03 20:45:20.629870 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:20.629717 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n" event={"ID":"3bf22ba3-3e73-4cc9-acd6-467414ab7896","Type":"ContainerDied","Data":"73d8d4b751a46c38d8588fcef7fdd2bdb525a7bf6cfff486be05498792277136"} Nov 03 20:45:20.629870 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:20.629785 5111 scope.go:122] "RemoveContainer" containerID="3c1ef500c11902e34b8f126210dc0cd1dd8752f022428d55eb45dac64bf75c76" Nov 03 20:45:20.669877 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:20.669829 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n"] Nov 03 20:45:20.673169 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:20.673138 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-ffs5n"] Nov 03 20:45:22.183234 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:22.183196 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf22ba3-3e73-4cc9-acd6-467414ab7896" path="/var/lib/kubelet/pods/3bf22ba3-3e73-4cc9-acd6-467414ab7896/volumes" Nov 03 20:45:24.657598 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:24.657566 5111 generic.go:356] "Generic (PLEG): container finished" podID="b9cdc815-0afe-469b-b8da-19b5c92cfd79" containerID="d2d47258251bd352911dfcc298dba7456bb06e19a0a9e5e0cf932649d8eb1efd" exitCode=0 Nov 03 20:45:24.658074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:24.657652 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" event={"ID":"b9cdc815-0afe-469b-b8da-19b5c92cfd79","Type":"ContainerDied","Data":"d2d47258251bd352911dfcc298dba7456bb06e19a0a9e5e0cf932649d8eb1efd"} Nov 03 20:45:24.658119 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:24.658094 5111 scope.go:122] "RemoveContainer" containerID="d2d47258251bd352911dfcc298dba7456bb06e19a0a9e5e0cf932649d8eb1efd" Nov 03 20:45:24.659440 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:24.659412 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:45:24.659587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:24.659482 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:45:24.659587 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:24.659514 5111 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:45:25.676255 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:25.676220 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" event={"ID":"b9cdc815-0afe-469b-b8da-19b5c92cfd79","Type":"ContainerStarted","Data":"9760a084eac4760c6cbee377fb40835a62f0961ad2e54a3b925bba23f7108258"} Nov 03 20:45:25.967620 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:25.963013 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:25.976564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:25.972683 5111 patch_prober.go:29] interesting pod/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Nov 03 20:45:25.976564 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:25.972752 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" podUID="b9cdc815-0afe-469b-b8da-19b5c92cfd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Nov 03 20:45:26.566502 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:26.566468 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:26.964989 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:26.964940 5111 patch_prober.go:29] interesting pod/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Nov 03 20:45:26.966357 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:26.966323 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" podUID="b9cdc815-0afe-469b-b8da-19b5c92cfd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Nov 03 20:45:27.969565 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:27.965892 5111 patch_prober.go:29] interesting pod/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Nov 03 20:45:27.969565 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:27.965959 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" podUID="b9cdc815-0afe-469b-b8da-19b5c92cfd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Nov 03 20:45:28.961675 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:28.961630 5111 patch_prober.go:29] interesting pod/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Nov 03 20:45:28.961994 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:28.961963 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" podUID="b9cdc815-0afe-469b-b8da-19b5c92cfd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Nov 03 20:45:29.969594 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:29.969511 5111 patch_prober.go:29] interesting pod/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Nov 03 20:45:29.970093 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:29.969646 5111 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" podUID="b9cdc815-0afe-469b-b8da-19b5c92cfd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Nov 03 20:45:30.965802 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:30.965771 5111 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:30.967163 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:30.967141 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:35.057635 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:45:35.057593 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:45:39.389114 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:39.389079 5111 scope.go:122] "RemoveContainer" containerID="58c24c1fa05451cb579a89bdaba6523b9f1e067efc736db55e930c0b94fc1650" Nov 03 20:45:43.321418 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:43.321379 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4"] Nov 03 20:45:43.322087 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:43.321721 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" podUID="b9cdc815-0afe-469b-b8da-19b5c92cfd79" containerName="istio-proxy" containerID="cri-o://9760a084eac4760c6cbee377fb40835a62f0961ad2e54a3b925bba23f7108258" gracePeriod=30 Nov 03 20:45:45.969342 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:45.969301 5111 patch_prober.go:29] interesting pod/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:45:45.969885 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:45.969361 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" podUID="b9cdc815-0afe-469b-b8da-19b5c92cfd79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:45:48.593505 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.593440 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:48.732984 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.732942 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-envoy\") pod \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " Nov 03 20:45:48.733202 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.732996 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-token\" (UniqueName: \"kubernetes.io/projected/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-token\") pod \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " Nov 03 20:45:48.733202 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.733026 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-podinfo\") pod \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " Nov 03 20:45:48.733202 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.733103 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-workload-certs\") pod \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " Nov 03 20:45:48.733202 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.733167 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-workload-socket\") pod \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " Nov 03 20:45:48.733477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.733214 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9cdc815-0afe-469b-b8da-19b5c92cfd79-kube-api-access-2997k\" (UniqueName: \"kubernetes.io/projected/b9cdc815-0afe-469b-b8da-19b5c92cfd79-kube-api-access-2997k\") pod \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " Nov 03 20:45:48.733477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.733245 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-data\") pod \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " Nov 03 20:45:48.733477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.733303 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-credential-socket\") pod \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " Nov 03 20:45:48.733477 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.733357 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istiod-ca-cert\") pod \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\" (UID: \"b9cdc815-0afe-469b-b8da-19b5c92cfd79\") " Nov 03 20:45:48.733755 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.733616 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-workload-socket" pod "b9cdc815-0afe-469b-b8da-19b5c92cfd79" (UID: "b9cdc815-0afe-469b-b8da-19b5c92cfd79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:45:48.733755 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.733650 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-data" pod "b9cdc815-0afe-469b-b8da-19b5c92cfd79" (UID: "b9cdc815-0afe-469b-b8da-19b5c92cfd79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:45:48.733887 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.733843 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-workload-certs" pod "b9cdc815-0afe-469b-b8da-19b5c92cfd79" (UID: "b9cdc815-0afe-469b-b8da-19b5c92cfd79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:45:48.733887 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.733876 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-credential-socket" pod "b9cdc815-0afe-469b-b8da-19b5c92cfd79" (UID: "b9cdc815-0afe-469b-b8da-19b5c92cfd79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:45:48.734029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.734006 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istiod-ca-cert" pod "b9cdc815-0afe-469b-b8da-19b5c92cfd79" (UID: "b9cdc815-0afe-469b-b8da-19b5c92cfd79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:48.734095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.734030 5111 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-credential-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.734095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.734055 5111 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-workload-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.734095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.734070 5111 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-workload-socket\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.734095 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.734084 5111 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-data\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.735549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.735515 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-envoy" pod "b9cdc815-0afe-469b-b8da-19b5c92cfd79" (UID: "b9cdc815-0afe-469b-b8da-19b5c92cfd79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:45:48.735669 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.735547 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9cdc815-0afe-469b-b8da-19b5c92cfd79-kube-api-access-2997k" pod "b9cdc815-0afe-469b-b8da-19b5c92cfd79" (UID: "b9cdc815-0afe-469b-b8da-19b5c92cfd79"). InnerVolumeSpecName "kube-api-access-2997k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:48.735669 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.735576 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-token" pod "b9cdc815-0afe-469b-b8da-19b5c92cfd79" (UID: "b9cdc815-0afe-469b-b8da-19b5c92cfd79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:48.735669 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.735643 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-podinfo" pod "b9cdc815-0afe-469b-b8da-19b5c92cfd79" (UID: "b9cdc815-0afe-469b-b8da-19b5c92cfd79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:45:48.826023 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.825991 5111 generic.go:356] "Generic (PLEG): container finished" podID="b9cdc815-0afe-469b-b8da-19b5c92cfd79" containerID="9760a084eac4760c6cbee377fb40835a62f0961ad2e54a3b925bba23f7108258" exitCode=0 Nov 03 20:45:48.826219 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.826042 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" Nov 03 20:45:48.826219 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.826072 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" event={"ID":"b9cdc815-0afe-469b-b8da-19b5c92cfd79","Type":"ContainerDied","Data":"9760a084eac4760c6cbee377fb40835a62f0961ad2e54a3b925bba23f7108258"} Nov 03 20:45:48.826219 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.826128 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4" event={"ID":"b9cdc815-0afe-469b-b8da-19b5c92cfd79","Type":"ContainerDied","Data":"8eda643b59d8efe4e1129ecb4db504e9387fa08e197cca56c6d6558a6121e393"} Nov 03 20:45:48.826219 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.826152 5111 scope.go:122] "RemoveContainer" containerID="9760a084eac4760c6cbee377fb40835a62f0961ad2e54a3b925bba23f7108258" Nov 03 20:45:48.835156 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.835101 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2997k\" (UniqueName: \"kubernetes.io/projected/b9cdc815-0afe-469b-b8da-19b5c92cfd79-kube-api-access-2997k\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.835156 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.835136 5111 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istiod-ca-cert\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.835156 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.835153 5111 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-envoy\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.835366 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.835168 5111 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-token\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.835366 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.835181 5111 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9cdc815-0afe-469b-b8da-19b5c92cfd79-istio-podinfo\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.837528 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.837503 5111 scope.go:122] "RemoveContainer" containerID="d2d47258251bd352911dfcc298dba7456bb06e19a0a9e5e0cf932649d8eb1efd" Nov 03 20:45:48.846685 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.846664 5111 scope.go:122] "RemoveContainer" containerID="9760a084eac4760c6cbee377fb40835a62f0961ad2e54a3b925bba23f7108258" Nov 03 20:45:48.847008 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:45:48.846987 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9760a084eac4760c6cbee377fb40835a62f0961ad2e54a3b925bba23f7108258\": container with ID starting with 9760a084eac4760c6cbee377fb40835a62f0961ad2e54a3b925bba23f7108258 not found: ID does not exist" containerID="9760a084eac4760c6cbee377fb40835a62f0961ad2e54a3b925bba23f7108258" Nov 03 20:45:48.847059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.847019 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9760a084eac4760c6cbee377fb40835a62f0961ad2e54a3b925bba23f7108258"} err="failed to get container status \"9760a084eac4760c6cbee377fb40835a62f0961ad2e54a3b925bba23f7108258\": rpc error: code = NotFound desc = could not find container \"9760a084eac4760c6cbee377fb40835a62f0961ad2e54a3b925bba23f7108258\": container with ID starting with 9760a084eac4760c6cbee377fb40835a62f0961ad2e54a3b925bba23f7108258 not found: ID does not exist" Nov 03 20:45:48.847059 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.847037 5111 scope.go:122] "RemoveContainer" containerID="d2d47258251bd352911dfcc298dba7456bb06e19a0a9e5e0cf932649d8eb1efd" Nov 03 20:45:48.847310 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:45:48.847289 5111 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d47258251bd352911dfcc298dba7456bb06e19a0a9e5e0cf932649d8eb1efd\": container with ID starting with d2d47258251bd352911dfcc298dba7456bb06e19a0a9e5e0cf932649d8eb1efd not found: ID does not exist" containerID="d2d47258251bd352911dfcc298dba7456bb06e19a0a9e5e0cf932649d8eb1efd" Nov 03 20:45:48.847374 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.847320 5111 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d47258251bd352911dfcc298dba7456bb06e19a0a9e5e0cf932649d8eb1efd"} err="failed to get container status \"d2d47258251bd352911dfcc298dba7456bb06e19a0a9e5e0cf932649d8eb1efd\": rpc error: code = NotFound desc = could not find container \"d2d47258251bd352911dfcc298dba7456bb06e19a0a9e5e0cf932649d8eb1efd\": container with ID starting with d2d47258251bd352911dfcc298dba7456bb06e19a0a9e5e0cf932649d8eb1efd not found: ID does not exist" Nov 03 20:45:48.853234 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.853203 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4"] Nov 03 20:45:48.857479 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:48.857448 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-364642e1-openshift-default-f6946bcc9-2k2w4"] Nov 03 20:45:49.342570 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.342493 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd"] Nov 03 20:45:49.343264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.343235 5111 cpu_manager.go:414] "containerMap: removing container" podUID="3bf22ba3-3e73-4cc9-acd6-467414ab7896" containerName="cleanup" Nov 03 20:45:49.343264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.343263 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf22ba3-3e73-4cc9-acd6-467414ab7896" containerName="cleanup" Nov 03 20:45:49.343491 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.343295 5111 cpu_manager.go:414] "containerMap: removing container" podUID="b9cdc815-0afe-469b-b8da-19b5c92cfd79" containerName="istio-proxy" Nov 03 20:45:49.343491 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.343303 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9cdc815-0afe-469b-b8da-19b5c92cfd79" containerName="istio-proxy" Nov 03 20:45:49.343491 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.343318 5111 cpu_manager.go:414] "containerMap: removing container" podUID="28f1edfb-f711-475f-b517-0800f594d261" containerName="maas-api" Nov 03 20:45:49.343491 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.343327 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f1edfb-f711-475f-b517-0800f594d261" containerName="maas-api" Nov 03 20:45:49.343491 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.343355 5111 cpu_manager.go:414] "containerMap: removing container" podUID="b9cdc815-0afe-469b-b8da-19b5c92cfd79" containerName="istio-proxy" Nov 03 20:45:49.343491 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.343362 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9cdc815-0afe-469b-b8da-19b5c92cfd79" containerName="istio-proxy" Nov 03 20:45:49.343491 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.343382 5111 cpu_manager.go:414] "containerMap: removing container" podUID="3bf22ba3-3e73-4cc9-acd6-467414ab7896" containerName="cleanup" Nov 03 20:45:49.343491 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.343389 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf22ba3-3e73-4cc9-acd6-467414ab7896" containerName="cleanup" Nov 03 20:45:49.343891 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.343526 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf22ba3-3e73-4cc9-acd6-467414ab7896" containerName="cleanup" Nov 03 20:45:49.343891 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.343590 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf22ba3-3e73-4cc9-acd6-467414ab7896" containerName="cleanup" Nov 03 20:45:49.343891 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.343603 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9cdc815-0afe-469b-b8da-19b5c92cfd79" containerName="istio-proxy" Nov 03 20:45:49.343891 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.343611 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f1edfb-f711-475f-b517-0800f594d261" containerName="maas-api" Nov 03 20:45:49.348944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.348919 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:45:49.356029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.355998 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-ca53c0-serving-cert\"" Nov 03 20:45:49.357337 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.357312 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd"] Nov 03 20:45:49.542470 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.542425 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec150dd-e071-40ae-9d42-f32ac18d3035-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd\" (UID: \"8ec150dd-e071-40ae-9d42-f32ac18d3035\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:45:49.542730 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.542484 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec150dd-e071-40ae-9d42-f32ac18d3035-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd\" (UID: \"8ec150dd-e071-40ae-9d42-f32ac18d3035\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:45:49.542730 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.542701 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbw75\" (UniqueName: \"kubernetes.io/projected/8ec150dd-e071-40ae-9d42-f32ac18d3035-kube-api-access-fbw75\") pod \"maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd\" (UID: \"8ec150dd-e071-40ae-9d42-f32ac18d3035\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:45:49.542844 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.542813 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec150dd-e071-40ae-9d42-f32ac18d3035-metrics-certs\") pod \"maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd\" (UID: \"8ec150dd-e071-40ae-9d42-f32ac18d3035\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:45:49.643910 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.643876 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec150dd-e071-40ae-9d42-f32ac18d3035-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd\" (UID: \"8ec150dd-e071-40ae-9d42-f32ac18d3035\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:45:49.644460 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.643966 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbw75\" (UniqueName: \"kubernetes.io/projected/8ec150dd-e071-40ae-9d42-f32ac18d3035-kube-api-access-fbw75\") pod \"maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd\" (UID: \"8ec150dd-e071-40ae-9d42-f32ac18d3035\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:45:49.644460 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.644026 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec150dd-e071-40ae-9d42-f32ac18d3035-metrics-certs\") pod \"maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd\" (UID: \"8ec150dd-e071-40ae-9d42-f32ac18d3035\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:45:49.644460 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.644075 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec150dd-e071-40ae-9d42-f32ac18d3035-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd\" (UID: \"8ec150dd-e071-40ae-9d42-f32ac18d3035\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:45:49.644899 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.644875 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec150dd-e071-40ae-9d42-f32ac18d3035-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd\" (UID: \"8ec150dd-e071-40ae-9d42-f32ac18d3035\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:45:49.646697 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.646674 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec150dd-e071-40ae-9d42-f32ac18d3035-metrics-certs\") pod \"maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd\" (UID: \"8ec150dd-e071-40ae-9d42-f32ac18d3035\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:45:49.646838 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.646814 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec150dd-e071-40ae-9d42-f32ac18d3035-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd\" (UID: \"8ec150dd-e071-40ae-9d42-f32ac18d3035\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:45:49.652669 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.652641 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbw75\" (UniqueName: \"kubernetes.io/projected/8ec150dd-e071-40ae-9d42-f32ac18d3035-kube-api-access-fbw75\") pod \"maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd\" (UID: \"8ec150dd-e071-40ae-9d42-f32ac18d3035\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:45:49.669284 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.669248 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:45:49.949125 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:49.949099 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd"] Nov 03 20:45:49.952288 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:45:49.952250 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec150dd_e071_40ae_9d42_f32ac18d3035.slice/crio-8e8555c58987ac4ff81af342bdf42621bf17348d36bf786fbd60fdccbc5abadd WatchSource:0}: Error finding container 8e8555c58987ac4ff81af342bdf42621bf17348d36bf786fbd60fdccbc5abadd: Status 404 returned error can't find the container with id 8e8555c58987ac4ff81af342bdf42621bf17348d36bf786fbd60fdccbc5abadd Nov 03 20:45:50.189966 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:50.189928 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9cdc815-0afe-469b-b8da-19b5c92cfd79" path="/var/lib/kubelet/pods/b9cdc815-0afe-469b-b8da-19b5c92cfd79/volumes" Nov 03 20:45:50.840549 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:50.840493 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" event={"ID":"8ec150dd-e071-40ae-9d42-f32ac18d3035","Type":"ContainerStarted","Data":"27227655bb69b0db504621249e374d3e411182f1446db9da7963d68674b718cc"} Nov 03 20:45:50.840999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:50.840577 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:45:50.840999 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:50.840602 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" event={"ID":"8ec150dd-e071-40ae-9d42-f32ac18d3035","Type":"ContainerStarted","Data":"8e8555c58987ac4ff81af342bdf42621bf17348d36bf786fbd60fdccbc5abadd"} Nov 03 20:45:50.859754 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:50.859696 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" podStartSLOduration=1.464662866 podStartE2EDuration="1.859679317s" totalImagesPullingTime="395.016451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:50.856497511 +0000 UTC m=+3980.818875950" watchObservedRunningTime="2026-11-03 20:45:50.859679317 +0000 UTC m=+3980.822057756" Nov 03 20:45:56.852638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:45:56.852605 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:46:05.063375 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:46:05.063327 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:46:35.057906 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:46:35.057812 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:46:48.030937 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:46:48.030897 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64"] Nov 03 20:46:48.034556 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:46:48.034507 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jbf64"] Nov 03 20:46:48.189315 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:46:48.189272 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb95ec1-e019-477c-bbec-0ce699cd3fac" path="/var/lib/kubelet/pods/efb95ec1-e019-477c-bbec-0ce699cd3fac/volumes" Nov 03 20:47:05.058180 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:47:05.058136 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:47:07.024185 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:07.024144 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66"] Nov 03 20:47:07.026089 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:07.026065 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-spn66"] Nov 03 20:47:08.181876 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:08.181841 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e29b5fc7-ca2a-4985-a1fe-7350a565f86f" path="/var/lib/kubelet/pods/e29b5fc7-ca2a-4985-a1fe-7350a565f86f/volumes" Nov 03 20:47:35.063076 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:47:35.063032 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:47:38.194299 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.194263 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd"] Nov 03 20:47:38.195190 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.195154 5111 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" podUID="8ec150dd-e071-40ae-9d42-f32ac18d3035" containerName="maas-api" containerID="cri-o://27227655bb69b0db504621249e374d3e411182f1446db9da7963d68674b718cc" gracePeriod=30 Nov 03 20:47:38.448306 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.448215 5111 generic.go:356] "Generic (PLEG): container finished" podID="8ec150dd-e071-40ae-9d42-f32ac18d3035" containerID="27227655bb69b0db504621249e374d3e411182f1446db9da7963d68674b718cc" exitCode=0 Nov 03 20:47:38.448473 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.448292 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" event={"ID":"8ec150dd-e071-40ae-9d42-f32ac18d3035","Type":"ContainerDied","Data":"27227655bb69b0db504621249e374d3e411182f1446db9da7963d68674b718cc"} Nov 03 20:47:38.512499 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.512467 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:47:38.631040 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.631005 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec150dd-e071-40ae-9d42-f32ac18d3035-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec150dd-e071-40ae-9d42-f32ac18d3035-openshift-service-ca\") pod \"8ec150dd-e071-40ae-9d42-f32ac18d3035\" (UID: \"8ec150dd-e071-40ae-9d42-f32ac18d3035\") " Nov 03 20:47:38.631236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.631100 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ec150dd-e071-40ae-9d42-f32ac18d3035-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec150dd-e071-40ae-9d42-f32ac18d3035-maas-api-tls\") pod \"8ec150dd-e071-40ae-9d42-f32ac18d3035\" (UID: \"8ec150dd-e071-40ae-9d42-f32ac18d3035\") " Nov 03 20:47:38.631236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.631155 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec150dd-e071-40ae-9d42-f32ac18d3035-kube-api-access-fbw75\" (UniqueName: \"kubernetes.io/projected/8ec150dd-e071-40ae-9d42-f32ac18d3035-kube-api-access-fbw75\") pod \"8ec150dd-e071-40ae-9d42-f32ac18d3035\" (UID: \"8ec150dd-e071-40ae-9d42-f32ac18d3035\") " Nov 03 20:47:38.631236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.631187 5111 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ec150dd-e071-40ae-9d42-f32ac18d3035-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec150dd-e071-40ae-9d42-f32ac18d3035-metrics-certs\") pod \"8ec150dd-e071-40ae-9d42-f32ac18d3035\" (UID: \"8ec150dd-e071-40ae-9d42-f32ac18d3035\") " Nov 03 20:47:38.631410 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.631373 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec150dd-e071-40ae-9d42-f32ac18d3035-openshift-service-ca" pod "8ec150dd-e071-40ae-9d42-f32ac18d3035" (UID: "8ec150dd-e071-40ae-9d42-f32ac18d3035"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:38.631657 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.631632 5111 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec150dd-e071-40ae-9d42-f32ac18d3035-openshift-service-ca\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:38.633343 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.633315 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ec150dd-e071-40ae-9d42-f32ac18d3035-metrics-certs" pod "8ec150dd-e071-40ae-9d42-f32ac18d3035" (UID: "8ec150dd-e071-40ae-9d42-f32ac18d3035"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:38.633343 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.633324 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec150dd-e071-40ae-9d42-f32ac18d3035-kube-api-access-fbw75" pod "8ec150dd-e071-40ae-9d42-f32ac18d3035" (UID: "8ec150dd-e071-40ae-9d42-f32ac18d3035"). InnerVolumeSpecName "kube-api-access-fbw75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:38.633492 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.633448 5111 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ec150dd-e071-40ae-9d42-f32ac18d3035-maas-api-tls" pod "8ec150dd-e071-40ae-9d42-f32ac18d3035" (UID: "8ec150dd-e071-40ae-9d42-f32ac18d3035"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:38.732785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.732688 5111 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec150dd-e071-40ae-9d42-f32ac18d3035-maas-api-tls\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:38.732785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.732724 5111 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbw75\" (UniqueName: \"kubernetes.io/projected/8ec150dd-e071-40ae-9d42-f32ac18d3035-kube-api-access-fbw75\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:38.732785 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:38.732737 5111 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec150dd-e071-40ae-9d42-f32ac18d3035-metrics-certs\") on node \"ip-10-0-134-178.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:39.454987 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:39.454959 5111 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" Nov 03 20:47:39.455424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:39.454957 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd" event={"ID":"8ec150dd-e071-40ae-9d42-f32ac18d3035","Type":"ContainerDied","Data":"8e8555c58987ac4ff81af342bdf42621bf17348d36bf786fbd60fdccbc5abadd"} Nov 03 20:47:39.455424 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:39.455098 5111 scope.go:122] "RemoveContainer" containerID="27227655bb69b0db504621249e374d3e411182f1446db9da7963d68674b718cc" Nov 03 20:47:39.481750 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:39.481655 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd"] Nov 03 20:47:39.483638 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:39.483612 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ca53c0-56f7467ff-5dcgd"] Nov 03 20:47:40.182751 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:40.182719 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ec150dd-e071-40ae-9d42-f32ac18d3035" path="/var/lib/kubelet/pods/8ec150dd-e071-40ae-9d42-f32ac18d3035/volumes" Nov 03 20:47:50.273634 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:50.271711 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2z5gd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:47:50.273634 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:50.271852 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2z5gd" podUID="130f1293-a57b-49b5-8e6a-471751b565f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:50.273634 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:50.272355 5111 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-lc2bv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:47:50.273634 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:47:50.272411 5111 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-lc2bv" podUID="37ddf8c6-464e-4a46-85f9-4cd0161e5b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:48:05.057561 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:48:05.057504 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:48:34.842557 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:34.841564 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77c9c58d6b-22kjj_99039821-d164-41f5-b9c3-85fd95744441/authorino/0.log" Nov 03 20:48:35.215682 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:48:35.215609 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:48:39.443054 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:39.443013 5111 scope.go:122] "RemoveContainer" containerID="66ac2cdc75f7352ab2ab13aa9c294f54a3e06ce05ef686b91dac2771293ea5e8" Nov 03 20:48:39.452380 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:39.452362 5111 scope.go:122] "RemoveContainer" containerID="834cb6dfd5d76c3434aaa5aa2c13a835e9aa46d5e152c2041c0a9ff20273c683" Nov 03 20:48:44.992654 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:44.992619 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-656bfc8bbd-llxlr_2716609d-ca4a-4bdf-8d24-3f06060a01e4/manager/0.log" Nov 03 20:48:47.165608 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:47.165579 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-876c878b5-lxzrv_d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3/maas-api/0.log" Nov 03 20:48:47.737933 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:47.737897 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-gc4bf_fb2f72d9-57be-4eb0-8e5a-f031981fbccb/revoke-keys/0.log" Nov 03 20:48:47.913317 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:47.913288 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-nsv5g_f598fc63-845a-43c0-84d1-9189ac70bcae/revoke-keys/0.log" Nov 03 20:48:48.263955 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:48.263923 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-85e32503-h8cgr_6aa9d26b-9326-4736-b334-fcf5d6e630f3/revoke-keys/0.log" Nov 03 20:48:50.098007 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:50.097971 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77c9c58d6b-22kjj_99039821-d164-41f5-b9c3-85fd95744441/authorino/0.log" Nov 03 20:48:51.285887 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:51.285860 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-5rjzk_2e8c119f-1121-4c27-ac2e-f625ff474965/manager/0.log" Nov 03 20:48:52.135227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:52.135198 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7c689579c-pv54h_002888c8-c3b7-4ae8-942a-e16e9263222b/istio-proxy/0.log" Nov 03 20:48:53.028868 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:53.028835 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2z5gd_130f1293-a57b-49b5-8e6a-471751b565f9/istio-proxy/1.log" Nov 03 20:48:53.402423 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:53.402394 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-8bcwd_ea1836fd-1444-44ab-831c-165ef7470a5c/istio-proxy/0.log" Nov 03 20:48:53.780490 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:53.780410 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-h7vtl_d969be31-5822-40ef-be6f-c6627bfa5956/istio-proxy/0.log" Nov 03 20:48:54.294828 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:54.294800 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-lc2bv_37ddf8c6-464e-4a46-85f9-4cd0161e5b6c/istio-proxy/1.log" Nov 03 20:48:54.644019 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:54.643984 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f_65ed7c10-e620-4856-b654-866ee1fc916f/istio-proxy/0.log" Nov 03 20:48:55.155486 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:55.155451 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-855cd4cff-prjqj_9924fd9e-f7b6-4036-8584-22caa1de39b3/router/0.log" Nov 03 20:48:55.754347 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:55.754314 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2z5gd_130f1293-a57b-49b5-8e6a-471751b565f9/istio-proxy/0.log" Nov 03 20:48:56.280644 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:56.280611 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-lc2bv_37ddf8c6-464e-4a46-85f9-4cd0161e5b6c/istio-proxy/0.log" Nov 03 20:48:57.164236 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:57.164204 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-8x8xh_457e9dde-adec-4ec7-83bd-8d157b9a3362/main/0.log" Nov 03 20:48:57.326956 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:57.326924 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-c8tdm_989fbc44-448a-4608-904f-b265dcaccce5/main/0.log" Nov 03 20:48:57.673207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:57.673172 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6_1e292de6-b36f-481b-9d69-1961e7d8026b/main/0.log" Nov 03 20:48:57.853500 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:57.853474 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-rvrqd_e49d0823-94c9-43a3-9963-72ed46e78ae8/main/0.log" Nov 03 20:48:58.199165 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:58.199130 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r_e52a6c08-4959-4cf8-90a6-fe48e0dbd12b/main/0.log" Nov 03 20:48:59.076018 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:48:59.075989 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77c9c58d6b-22kjj_99039821-d164-41f5-b9c3-85fd95744441/authorino/0.log" Nov 03 20:49:05.055958 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:49:05.055913 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:49:08.113259 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:08.113221 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-656bfc8bbd-llxlr_2716609d-ca4a-4bdf-8d24-3f06060a01e4/manager/0.log" Nov 03 20:49:10.077314 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:10.077283 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-876c878b5-lxzrv_d8167fb5-7d3e-479c-a2e8-bd0ae3498ba3/maas-api/0.log" Nov 03 20:49:10.623546 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:10.623512 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-gc4bf_fb2f72d9-57be-4eb0-8e5a-f031981fbccb/revoke-keys/0.log" Nov 03 20:49:10.800070 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:10.800036 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-nsv5g_f598fc63-845a-43c0-84d1-9189ac70bcae/revoke-keys/0.log" Nov 03 20:49:11.141709 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:11.141678 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-85e32503-h8cgr_6aa9d26b-9326-4736-b334-fcf5d6e630f3/revoke-keys/0.log" Nov 03 20:49:12.579786 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:12.579700 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77c9c58d6b-22kjj_99039821-d164-41f5-b9c3-85fd95744441/authorino/0.log" Nov 03 20:49:13.728074 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:13.728036 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-5rjzk_2e8c119f-1121-4c27-ac2e-f625ff474965/manager/0.log" Nov 03 20:49:14.271646 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:14.271613 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7c689579c-pv54h_002888c8-c3b7-4ae8-942a-e16e9263222b/istio-proxy/0.log" Nov 03 20:49:15.020167 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:15.020128 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g"] Nov 03 20:49:15.022644 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:15.022619 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nsv5g"] Nov 03 20:49:15.160833 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:15.160806 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2z5gd_130f1293-a57b-49b5-8e6a-471751b565f9/istio-proxy/1.log" Nov 03 20:49:15.516043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:15.516009 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-8bcwd_ea1836fd-1444-44ab-831c-165ef7470a5c/istio-proxy/0.log" Nov 03 20:49:15.856455 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:15.856376 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-h7vtl_d969be31-5822-40ef-be6f-c6627bfa5956/istio-proxy/0.log" Nov 03 20:49:16.181011 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:16.180979 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f598fc63-845a-43c0-84d1-9189ac70bcae" path="/var/lib/kubelet/pods/f598fc63-845a-43c0-84d1-9189ac70bcae/volumes" Nov 03 20:49:16.393714 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:16.393686 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-lc2bv_37ddf8c6-464e-4a46-85f9-4cd0161e5b6c/istio-proxy/1.log" Nov 03 20:49:16.750967 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:16.750930 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f_65ed7c10-e620-4856-b654-866ee1fc916f/istio-proxy/0.log" Nov 03 20:49:17.288865 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:17.288824 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-855cd4cff-prjqj_9924fd9e-f7b6-4036-8584-22caa1de39b3/router/0.log" Nov 03 20:49:18.007861 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:18.007820 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-8x8xh_457e9dde-adec-4ec7-83bd-8d157b9a3362/main/0.log" Nov 03 20:49:18.184885 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:18.184859 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-c8tdm_989fbc44-448a-4608-904f-b265dcaccce5/main/0.log" Nov 03 20:49:18.525943 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:18.525902 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sz8d6_1e292de6-b36f-481b-9d69-1961e7d8026b/main/0.log" Nov 03 20:49:18.700729 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:18.700696 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-rvrqd_e49d0823-94c9-43a3-9963-72ed46e78ae8/main/0.log" Nov 03 20:49:19.044149 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:19.044111 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-dbd6r_e52a6c08-4959-4cf8-90a6-fe48e0dbd12b/main/0.log" Nov 03 20:49:31.091266 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:31.091239 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:49:31.096066 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:31.096043 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:49:31.096207 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:31.096160 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:49:31.101400 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:31.101382 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tzzd5_754b8c74-0816-45c8-92f6-95725095e262/ovn-acl-logging/0.log" Nov 03 20:49:32.021635 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:32.021594 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf"] Nov 03 20:49:32.022716 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:32.022690 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gc4bf"] Nov 03 20:49:32.182190 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:32.182159 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb2f72d9-57be-4eb0-8e5a-f031981fbccb" path="/var/lib/kubelet/pods/fb2f72d9-57be-4eb0-8e5a-f031981fbccb/volumes" Nov 03 20:49:35.057018 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:49:35.056984 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:49:53.020844 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:53.020796 5111 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr"] Nov 03 20:49:53.022572 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:53.022525 5111 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e32503-h8cgr"] Nov 03 20:49:54.181894 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:49:54.181862 5111 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa9d26b-9326-4736-b334-fcf5d6e630f3" path="/var/lib/kubelet/pods/6aa9d26b-9326-4736-b334-fcf5d6e630f3/volumes" Nov 03 20:50:05.055994 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:50:05.055952 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:50:35.057371 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:50:35.057330 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:50:39.488954 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:39.488916 5111 scope.go:122] "RemoveContainer" containerID="64b7a03bd4cbc048602b0f96619c291c14538a5edb9ff3bce5d9c59b30d5e156" Nov 03 20:50:39.497987 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:39.497960 5111 scope.go:122] "RemoveContainer" containerID="cd4c3edcdb2aadad1959bb56074636ade1885d497bd12beea1d1772a6d9e4da2" Nov 03 20:50:47.464043 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:47.463970 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p4t7z_edae5b46-8158-4dc8-91c9-06ed65126db9/global-pull-secret-syncer/0.log" Nov 03 20:50:47.503609 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:47.503583 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-d5qxk_51c48ea1-6b42-4ec0-ae95-bf81b913e901/connection-checker/0.log" Nov 03 20:50:47.529431 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:47.529403 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-knv7j_d2e3f7cd-e4ba-44c3-9ed5-2508bb92c041/connection-checker/0.log" Nov 03 20:50:47.556493 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:47.556466 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-x9cw5_257528a3-a0f4-4a07-bba0-67561f8ee205/connection-checker/0.log" Nov 03 20:50:47.668755 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:47.668710 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f4x7r_19cae502-56f5-4ec2-9867-205025a68d83/konnectivity-agent/0.log" Nov 03 20:50:47.787147 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:47.787073 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-178.us-east-2.compute.internal_5cf8bc898cbd908afece93ec1ad60d21/haproxy/0.log" Nov 03 20:50:54.626708 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:54.626671 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77c9c58d6b-22kjj_99039821-d164-41f5-b9c3-85fd95744441/authorino/0.log" Nov 03 20:50:55.013600 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:55.013501 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-5rjzk_2e8c119f-1121-4c27-ac2e-f625ff474965/manager/0.log" Nov 03 20:50:56.447049 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.447009 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-zx2m4_ca34d416-e238-48cb-a507-953f78fead22/cluster-monitoring-operator/0.log" Nov 03 20:50:56.617098 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.617068 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8sczc_83bba5dc-5ae2-4fab-a021-d828309dd462/node-exporter/0.log" Nov 03 20:50:56.637523 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.637496 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8sczc_83bba5dc-5ae2-4fab-a021-d828309dd462/kube-rbac-proxy/0.log" Nov 03 20:50:56.660670 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.660648 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8sczc_83bba5dc-5ae2-4fab-a021-d828309dd462/init-textfile/0.log" Nov 03 20:50:56.750327 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.750255 5111 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn"] Nov 03 20:50:56.750792 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.750776 5111 cpu_manager.go:414] "containerMap: removing container" podUID="8ec150dd-e071-40ae-9d42-f32ac18d3035" containerName="maas-api" Nov 03 20:50:56.750852 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.750793 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec150dd-e071-40ae-9d42-f32ac18d3035" containerName="maas-api" Nov 03 20:50:56.750852 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.750813 5111 cpu_manager.go:414] "containerMap: removing container" podUID="3bf22ba3-3e73-4cc9-acd6-467414ab7896" containerName="cleanup" Nov 03 20:50:56.750852 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.750819 5111 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf22ba3-3e73-4cc9-acd6-467414ab7896" containerName="cleanup" Nov 03 20:50:56.750967 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.750910 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf22ba3-3e73-4cc9-acd6-467414ab7896" containerName="cleanup" Nov 03 20:50:56.750967 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.750922 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9cdc815-0afe-469b-b8da-19b5c92cfd79" containerName="istio-proxy" Nov 03 20:50:56.750967 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.750934 5111 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec150dd-e071-40ae-9d42-f32ac18d3035" containerName="maas-api" Nov 03 20:50:56.754434 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.754408 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:56.761768 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.761740 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn"] Nov 03 20:50:56.851628 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.851596 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5pd2z\"/\"kube-root-ca.crt\"" Nov 03 20:50:56.851855 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.851691 5111 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5pd2z\"/\"openshift-service-ca.crt\"" Nov 03 20:50:56.851855 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.851744 5111 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5pd2z\"/\"default-dockercfg-pvhgd\"" Nov 03 20:50:56.864279 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.864251 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/065820e2-115f-4f95-be53-a94575e91677-podres\") pod \"perf-node-gather-daemonset-mxntn\" (UID: \"065820e2-115f-4f95-be53-a94575e91677\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:56.864405 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.864292 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/065820e2-115f-4f95-be53-a94575e91677-sys\") pod \"perf-node-gather-daemonset-mxntn\" (UID: \"065820e2-115f-4f95-be53-a94575e91677\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:56.864405 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.864367 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/065820e2-115f-4f95-be53-a94575e91677-lib-modules\") pod \"perf-node-gather-daemonset-mxntn\" (UID: \"065820e2-115f-4f95-be53-a94575e91677\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:56.864487 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.864425 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24phb\" (UniqueName: \"kubernetes.io/projected/065820e2-115f-4f95-be53-a94575e91677-kube-api-access-24phb\") pod \"perf-node-gather-daemonset-mxntn\" (UID: \"065820e2-115f-4f95-be53-a94575e91677\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:56.864523 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.864492 5111 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/065820e2-115f-4f95-be53-a94575e91677-proc\") pod \"perf-node-gather-daemonset-mxntn\" (UID: \"065820e2-115f-4f95-be53-a94575e91677\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:56.965264 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.965204 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/065820e2-115f-4f95-be53-a94575e91677-lib-modules\") pod \"perf-node-gather-daemonset-mxntn\" (UID: \"065820e2-115f-4f95-be53-a94575e91677\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:56.965465 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.965310 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24phb\" (UniqueName: \"kubernetes.io/projected/065820e2-115f-4f95-be53-a94575e91677-kube-api-access-24phb\") pod \"perf-node-gather-daemonset-mxntn\" (UID: \"065820e2-115f-4f95-be53-a94575e91677\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:56.965465 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.965376 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/065820e2-115f-4f95-be53-a94575e91677-proc\") pod \"perf-node-gather-daemonset-mxntn\" (UID: \"065820e2-115f-4f95-be53-a94575e91677\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:56.965465 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.965385 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/065820e2-115f-4f95-be53-a94575e91677-lib-modules\") pod \"perf-node-gather-daemonset-mxntn\" (UID: \"065820e2-115f-4f95-be53-a94575e91677\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:56.965678 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.965472 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/065820e2-115f-4f95-be53-a94575e91677-proc\") pod \"perf-node-gather-daemonset-mxntn\" (UID: \"065820e2-115f-4f95-be53-a94575e91677\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:56.965678 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.965584 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/065820e2-115f-4f95-be53-a94575e91677-podres\") pod \"perf-node-gather-daemonset-mxntn\" (UID: \"065820e2-115f-4f95-be53-a94575e91677\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:56.965678 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.965634 5111 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/065820e2-115f-4f95-be53-a94575e91677-sys\") pod \"perf-node-gather-daemonset-mxntn\" (UID: \"065820e2-115f-4f95-be53-a94575e91677\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:56.965820 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.965694 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/065820e2-115f-4f95-be53-a94575e91677-podres\") pod \"perf-node-gather-daemonset-mxntn\" (UID: \"065820e2-115f-4f95-be53-a94575e91677\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:56.965820 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.965744 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/065820e2-115f-4f95-be53-a94575e91677-sys\") pod \"perf-node-gather-daemonset-mxntn\" (UID: \"065820e2-115f-4f95-be53-a94575e91677\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:56.973432 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:56.973410 5111 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24phb\" (UniqueName: \"kubernetes.io/projected/065820e2-115f-4f95-be53-a94575e91677-kube-api-access-24phb\") pod \"perf-node-gather-daemonset-mxntn\" (UID: \"065820e2-115f-4f95-be53-a94575e91677\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:57.070393 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:57.070307 5111 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:57.192944 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:57.192920 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-rs9bf_98e50f06-c94a-4d3c-a291-a85f0d18e564/prometheus-operator-admission-webhook/0.log" Nov 03 20:50:57.347317 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:57.347286 5111 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn"] Nov 03 20:50:57.350011 ip-10-0-134-178 kubenswrapper[5111]: W1103 20:50:57.349984 5111 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod065820e2_115f_4f95_be53_a94575e91677.slice/crio-420c8f48db80b19099d11f4f150560db0bc82be4687736d25f71a31d38acdf60 WatchSource:0}: Error finding container 420c8f48db80b19099d11f4f150560db0bc82be4687736d25f71a31d38acdf60: Status 404 returned error can't find the container with id 420c8f48db80b19099d11f4f150560db0bc82be4687736d25f71a31d38acdf60 Nov 03 20:50:57.679493 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:57.679456 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" event={"ID":"065820e2-115f-4f95-be53-a94575e91677","Type":"ContainerStarted","Data":"1112ebb1dd4be2609df45c7c71b6a47e9da9fc239252f11e449b4bc869d7acf1"} Nov 03 20:50:57.680042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:57.679507 5111 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" event={"ID":"065820e2-115f-4f95-be53-a94575e91677","Type":"ContainerStarted","Data":"420c8f48db80b19099d11f4f150560db0bc82be4687736d25f71a31d38acdf60"} Nov 03 20:50:57.680042 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:57.679554 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:50:57.695619 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:57.695566 5111 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" podStartSLOduration=1.695528048 podStartE2EDuration="1.695528048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:57.693981616 +0000 UTC m=+4287.656360054" watchObservedRunningTime="2026-11-03 20:50:57.695528048 +0000 UTC m=+4287.657906486" Nov 03 20:50:58.584032 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:58.583999 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-sk4zs_b2503201-686a-4cf9-b5a1-977c87e6b717/networking-console-plugin/0.log" Nov 03 20:50:59.211046 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:59.211018 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/1.log" Nov 03 20:50:59.216320 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:59.216287 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-68j2z_73f2218a-88f7-4f26-98b3-2d6403bff02d/console-operator/2.log" Nov 03 20:50:59.749459 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:59.749418 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-69f57dcd5b-8j7f7_c53940d4-bf84-4e09-a5ee-63a02d237c09/console/0.log" Nov 03 20:50:59.788009 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:50:59.787967 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-6h5km_5211dbec-f5a3-4b26-ba64-171cb475f97c/download-server/0.log" Nov 03 20:51:00.454030 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:00.453983 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-txtq2_0fbfc914-8c97-4770-9ef6-6e553c4dcc58/volume-data-source-validator/0.log" Nov 03 20:51:01.595656 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:01.595615 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zdf6j_5101a457-b490-4538-a9f6-288014fa9264/dns/0.log" Nov 03 20:51:01.617929 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:01.617905 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zdf6j_5101a457-b490-4538-a9f6-288014fa9264/kube-rbac-proxy/0.log" Nov 03 20:51:01.727409 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:01.727381 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k666m_29f79e6b-20d5-4ba4-8737-068f0d82b1e6/dns-node-resolver/0.log" Nov 03 20:51:02.416068 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:02.416025 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c87ddc7f5-dqz46_a91cdbfb-3a68-43c4-b773-6a8e2f32d224/registry/0.log" Nov 03 20:51:02.512896 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:02.512862 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pvs99_e90fc007-32bc-43c5-99c6-7dbb3f9753d7/node-ca/0.log" Nov 03 20:51:03.537672 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:03.537639 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7c689579c-pv54h_002888c8-c3b7-4ae8-942a-e16e9263222b/istio-proxy/0.log" Nov 03 20:51:03.697215 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:03.697180 5111 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-mxntn" Nov 03 20:51:05.060281 ip-10-0-134-178 kubenswrapper[5111]: E1103 20:51:05.060244 5111 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-md5d2" containerName="extractor" Nov 03 20:51:05.853942 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:05.853911 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2z5gd_130f1293-a57b-49b5-8e6a-471751b565f9/istio-proxy/1.log" Nov 03 20:51:05.854217 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:05.854190 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2z5gd_130f1293-a57b-49b5-8e6a-471751b565f9/istio-proxy/0.log" Nov 03 20:51:05.910720 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:05.910687 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-8bcwd_ea1836fd-1444-44ab-831c-165ef7470a5c/istio-proxy/0.log" Nov 03 20:51:05.968142 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:05.968105 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-h7vtl_d969be31-5822-40ef-be6f-c6627bfa5956/istio-proxy/0.log" Nov 03 20:51:06.055397 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:06.055364 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-lc2bv_37ddf8c6-464e-4a46-85f9-4cd0161e5b6c/istio-proxy/1.log" Nov 03 20:51:06.055595 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:06.055524 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-lc2bv_37ddf8c6-464e-4a46-85f9-4cd0161e5b6c/istio-proxy/0.log" Nov 03 20:51:06.103585 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:06.103556 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f_65ed7c10-e620-4856-b654-866ee1fc916f/istio-proxy/0.log" Nov 03 20:51:06.190349 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:06.190323 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-855cd4cff-prjqj_9924fd9e-f7b6-4036-8584-22caa1de39b3/router/0.log" Nov 03 20:51:07.933406 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:07.933377 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7c689579c-pv54h_002888c8-c3b7-4ae8-942a-e16e9263222b/istio-proxy/0.log" Nov 03 20:51:08.699323 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:08.699294 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ggbpk_4734a1de-ee4e-42ab-b925-1a85f65b7d24/serve-healthcheck-canary/0.log" Nov 03 20:51:09.276224 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:09.276196 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-rs8wk_2266dada-9604-44f9-b488-511006cee7b2/insights-operator/0.log" Nov 03 20:51:09.277757 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:09.277737 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-rs8wk_2266dada-9604-44f9-b488-511006cee7b2/insights-operator/1.log" Nov 03 20:51:09.460029 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:09.460000 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-md5d2_d134b335-9e0e-41c9-a676-566a6ef55f5a/kube-rbac-proxy/0.log" Nov 03 20:51:09.483664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:09.483631 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-md5d2_d134b335-9e0e-41c9-a676-566a6ef55f5a/exporter/0.log" Nov 03 20:51:09.509479 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:09.509450 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-md5d2_d134b335-9e0e-41c9-a676-566a6ef55f5a/extractor/0.log" Nov 03 20:51:09.550242 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:09.550174 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-6x92p-7c5mm_2055fe6f-33ae-45f0-a7b0-7ccec390e4e5/insights-gathering/0.log" Nov 03 20:51:09.623227 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:09.623194 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2z5gd_130f1293-a57b-49b5-8e6a-471751b565f9/istio-proxy/1.log" Nov 03 20:51:09.623507 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:09.623489 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2z5gd_130f1293-a57b-49b5-8e6a-471751b565f9/istio-proxy/0.log" Nov 03 20:51:09.658602 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:09.658578 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-8bcwd_ea1836fd-1444-44ab-831c-165ef7470a5c/istio-proxy/0.log" Nov 03 20:51:09.695185 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:09.695155 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-h7vtl_d969be31-5822-40ef-be6f-c6627bfa5956/istio-proxy/0.log" Nov 03 20:51:09.747197 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:09.747167 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-lc2bv_37ddf8c6-464e-4a46-85f9-4cd0161e5b6c/istio-proxy/1.log" Nov 03 20:51:09.747429 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:09.747367 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-lc2bv_37ddf8c6-464e-4a46-85f9-4cd0161e5b6c/istio-proxy/0.log" Nov 03 20:51:09.775584 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:09.775561 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-74987c9b47-8gh6f_65ed7c10-e620-4856-b654-866ee1fc916f/istio-proxy/0.log" Nov 03 20:51:09.832664 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:09.832594 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-855cd4cff-prjqj_9924fd9e-f7b6-4036-8584-22caa1de39b3/router/0.log" Nov 03 20:51:11.899032 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:11.898994 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5749d45fbb-7cpx6_e0800588-b452-4a3d-ab60-e0a522174a33/manager/0.log" Nov 03 20:51:11.925359 ip-10-0-134-178 kubenswrapper[5111]: I1103 20:51:11.925335 5111 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5749d45fbb-7cpx6_e0800588-b452-4a3d-ab60-e0a522174a33/copy-manifests/0.log"